System

Abstract

A system is provided.SOLUTION: A system including means for uploading a material, means for analyzing content of the uploaded material, means for generating an explanatory scenario on the basis of the analyzed content, means for creating a narration sound and a visual material on the basis of the generated scenario, means for generating a moving image by integrating the narration sound and the visual material, means for previewing the generated moving image and receiving a correction instruction, and means for providing a finally generated moving image to a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The goal is to solve the problems of parents being unable to provide accurate answers to their children's questions, and of corporate employees finding it time-consuming and difficult to understand complex work manuals. Specifically, the goal is to provide accurate and easy-to-understand information quickly, saving time and clarifying knowledge. [Means for solving the problem]

[0005] In order to solve this problem, a system is provided that includes the following means: First, a means for a user to upload materials. Next, a means for a server to analyze the content of the uploaded materials. Next, a means for generating an explanatory scenario based on the analyzed content. Further, a means for creating narration audio and visual materials based on the generated scenario. Next, a means for generating a video by integrating the created narration audio and visual materials. Next, a means for previewing the generated video and accepting correction instructions from the user. Finally, a means for providing the final generated video to the user. By using such a system, it is possible to quickly provide accurate and visually easy-to-understand information.

[0006] "Materials" refers to documents and data files containing information such as books, dictionaries, picture books, and manuals that users upload to the system.

[0007] "Means for uploading" refers to the function or interface that allows a user to send materials from their own terminal to the system and store the materials on the server.

[0008] "Means of analysis" refers to the functions and processes for extracting the content of uploaded materials and reading textual information and structured data using natural language processing technology and OCR technology.

[0009] "Means for generating a scenario" refers to the function of creating scripts and guidelines that constitute the flow of commentary and the content of narration based on the content of the analyzed materials.

[0010] "Means for creating narration audio" refers to the process of using speech synthesis technology or software to convert the text of the generated scenario into speech.

[0011] "Means for creating visual materials" refers to the process of generating or obtaining images and video materials such as diagrams, illustrations, and icons to visually represent the explanatory content.

[0012] "Means for generating video" refers to the function of integrating narration audio and visual materials and creating a single integrated video file using a video editing engine.

[0013] "Means for previewing" refers to a function or interface that allows a user to check the generated video and receives feedback and correction instructions from the user.

[0014] "Means provided to users" refers to the function that allows users to download the final verified and corrected video or generate a sharing link. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention is a system that uploads materials, analyzes the contents, generates videos, and provides them to users. The specific program processing flow and system operation are described in detail below.

[0037] Uploading and Loading Materials

[0038] The user uploads the selected book, dictionary, picture book, or material (PDF, text file, etc.) to the system. Using the device's web browser, the user selects the material on the designated upload screen and clicks the upload button. This operation sends the material to the server.

[0039] Text analysis of materials

[0040] The server analyzes the uploaded document. It uses a PDF parser and text parser to extract the text data from the document. It then uses natural language processing (NLP) technology to extract keywords and important sentences from the text and analyze the structure of the document. If necessary, OCR (optical character recognition) technology is used to convert text in images into text.

[0041] Scenario generation based on the content of the materials

[0042] The server generates a commentary scenario based on the analyzed text data. This is the process of determining the flow of the video and defining the narration content and insertion points for necessary visual materials. Structured data is created, specifying the commentary order and detailed narration content for each section.

[0043] Voice synthesis and visual material creation

[0044] The server creates narration audio and visual materials based on the generated scenario. It uses Text-to-Speech (TTS) technology to convert text into audio and create the narration. At the same time, visual materials (illustrations, diagrams, etc.) are generated using image generation AI and image editing tools.

[0045] Video Generation

[0046] The server combines the narration audio with the visual material to generate the video. Using a video editing engine (such as FFmpeg or OpenShot), the material is placed on a timeline and transition effects and text overlays are added to create a visually appealing video. At this stage, the video is exported to its final form and saved in an appropriate format, such as MP4.

[0047] Preview and edit your video

[0048] The user can preview the generated video and check it on the browser. The server provides a preview function that enables this, and after the user plays the video and checks the content, they can provide feedback on any necessary corrections. Upon receiving the correction request, the server re-edits the relevant parts and updates the video.

[0049] Download or share the video

[0050] Users can download the final video or generate a sharing link to share it with others. The server generates a secure download link, allows users to save the video locally, and generates a viewing link for internal or external sharing, and also provides the ability to send an email or generate a QR code if desired.

[0051] As a concrete example, let's consider the case where a parent creates an explanatory video for their child about the planets of the solar system. The parent uploads a PDF of an elementary school science textbook and inputs it into the system. The server analyzes the material and extracts information about the planets in the solar system. An explanatory scenario is then generated, and narration audio and planet illustrations are created. These materials are then integrated into a video. The parent can use the preview function to check the video and make any final corrections. After the corrections are complete, the parent can download the final video and show it to their child.

[0052] Through this series of steps, users can easily provide accurate information in video format, saving a great deal of time and effort. By using this system, information can be transmitted quickly and effectively.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] The user uploads the selected material (PDF, text file, etc.) to the system by opening a web browser on the terminal, selecting the material on the system's upload screen, and clicking the upload button.

[0056] Step 2:

[0057] The server receives and stores the uploaded file, first checking the file format and preparing to parse the content using a PDF parser or text parser.

[0058] Step 3:

[0059] The server analyzes the content of the document. In the case of a PDF, it uses a PDF parser to extract the text data and OCR technology to convert the characters in the image into text. In the case of a text file, it reads the text data directly. Next, it uses natural language processing (NLP) technology to extract key information (keywords, important sentences, paragraphs, etc.).

[0060] Step 4:

[0061] The server generates a commentary scenario based on the analyzed text data. The scenario includes the flow of the commentary, narration content, important points, and points where visual materials should be inserted. The scenario is organized as structured data.

[0062] Step 5:

[0063] The server creates narration audio and visual materials based on the scenario. It uses text-to-speech (TTS) technology to convert the scenario text into narration audio. At the same time, it uses image generation AI and image editing tools to generate the necessary visual materials, such as illustrations and diagrams.

[0064] Step 6:

[0065] The server generates a video by integrating the narration audio and visual materials. Using a video editing engine (e.g., FFmpeg or OpenShot), the audio files and visual materials are placed on a timeline. Furthermore, transition effects and text overlays are added to improve the visual quality of the video.

[0066] Step 7:

[0067] The server provides the generated video so that the user can preview it. The user can check the video in the browser and check whether the content matches the user's intention. If necessary, the user can specify corrections and provide feedback.

[0068] Step 8:

[0069] The server receives correction instructions from the user, re-edits the relevant parts, and provides the re-edited video for preview again so that the user can make a final check.

[0070] Step 9:

[0071] The server then provides the final video that the user has approved. The user can then download the video or generate a sharing link to share it with others. The server generates secure download links and viewing links, and also provides email and QR code generation functions as needed.

[0072] By following the above steps, users can easily create and quickly provide accurate and easy-to-understand instructional videos based on materials of their own choosing. This system can be used for a wide range of purposes, such as home education and creating business manuals in companies.

[0073] Example 1

[0074] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0075] Conventional document analysis systems often lack sufficient methods for efficiently understanding the content of documents. In particular, it is difficult to quickly and accurately analyze vast amounts of data and provide the information in a format that is easy for users to understand, which requires time and effort. Furthermore, simply providing the content of documents in text format makes it difficult to convert it into a visually understandable format, making it ineffective for use in education, presentations, and other applications. Therefore, there is a growing demand for a system that can analyze uploaded documents and provide information in video format based on the results.

[0076] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0077] In this invention, the server includes means for uploading materials, means for analyzing the content of the uploaded materials, means for generating an explanatory scenario based on the analyzed content, means for creating a narration voice and visual materials based on the generated scenario, means for generating a video by integrating the narration voice and visual materials, means for previewing the generated video and accepting correction instructions, and means for providing the final generated video to the user. This makes it possible to analyze the content of the uploaded materials and provide the information quickly and efficiently in a video format that is easy for the user to understand.

[0078] The "means for uploading materials" is a function for transmitting materials selected by the user to the system and storing them on the server.

[0079] "Means for analyzing the content of uploaded materials" refers to a function for extracting text data and other information from uploaded materials and converting that information into an easily understandable format.

[0080] The "means for generating an explanatory scenario" is a function that creates a scenario for determining the structure of the explanatory video and the content of the narration based on the content of the analyzed materials.

[0081] "Means for creating narration audio and visual materials" refers to a function for creating narration audio and visual materials using voice synthesis technology and image generation technology in accordance with the generated scenario.

[0082] "Means for integrating narration audio and visual materials to generate video" refers to a function that combines the created audio and visual materials on a timeline and outputs them as a completed video.

[0083] The "means for previewing the generated video and accepting correction instructions" is a function that allows the user to check the generated video and provide feedback on parts that need correction.

[0084] The "means for providing the final video to the user" is a function that allows the user to download the finalized video or generate a link that allows the user to share it with other people.

[0085] "Natural language processing technology" is a method for analyzing text data, extracting keywords and important sentences from it, and understanding the content of the material.

[0086] "Speech synthesis technology" is a technology that generates synthetic speech based on text data.

[0087] "Image generation technology" refers to the technology of creating visual materials based on text or other data.

[0088] A "video editing engine" is software or a system for generating and editing videos by combining narration audio and visual materials.

[0089] The present invention provides a system for uploading materials, analyzing the contents of the materials, generating videos, and providing the videos to users. Specific embodiments will be described in detail below.

[0090] Uploading and Loading Materials

[0091] The user accesses the system's upload screen using a web browser on their device and uploads the selected materials (PDF, text files, etc.). The user clicks the upload button, which sends the materials to the server.

[0092] Analysis of data

[0093] The server receives the uploaded documents and temporarily stores them in storage. It then uses a PDF parser (pdfminer.six) to extract text data from the PDF file. For text files, it directly reads the content. It then uses natural language processing (NLP) technology (NLTK) to extract keywords and important sentences from the text and analyze the structure of the document. If necessary, it also converts text within images into text using OCR (optical character recognition) technology (Tesseract OCR).

[0094] Generate a scenario

[0095] The server generates an explanatory scenario based on the analyzed text data. This scenario includes the flow of the video, the narration content, and insertion points for visual materials. To generate the scenario, a generative AI model is used and a prompt is entered. For example, the prompt might read, "Please create a scenario that explains the planets in the solar system for elementary school science classes. I have uploaded a PDF of a science textbook as a reference. Based on this reference, please generate a scenario that includes the characteristics, relative positions, and important information about each planet."

[0096] Voice synthesis and visual material creation

[0097] Based on the generated scenario, the server uses text-to-speech (TTS) technology (Google Text-to-Speech API) to convert text into speech and create the narration. At the same time, it uses image generation AI (e.g., DALL-E) and image editing tools (Adobe Photoshop) to create the necessary visual materials.

[0098] Video generation

[0099] The server integrates the created narration audio with the visual material, and then uses a video editing engine (e.g., FFmpeg) to place the material on a timeline, adding transition effects and text overlays to create a visually appealing video.

[0100] Preview and edit your video

[0101] The user can preview the generated video on the browser and check the content. The server provides a preview function and gives feedback to the user on any corrections that need to be made. The server receives the correction request, re-edits the relevant parts, and updates the video.

[0102] Video provision

[0103] Users can then download the final video or share it with others. The server generates a secure download link and provides it to the user. It can also generate a sharing link for viewing, which can be sent via email or generated as a QR code.

[0104] As a specific example, if a parent wants to create an explanatory video for their child about the planets in the solar system, they upload a PDF of a science textbook to the system. The server analyzes the material and extracts information about the planets in the solar system. The generative AI model then generates a scenario and uses speech synthesis and image generation technologies to create narration audio and illustrations of the planets. These are then integrated to generate a video, and the user (parent) can use the preview function to check the content, make any necessary corrections, and then download or share the final video.

[0105] This system allows users to easily create high-quality videos for educational and presentation purposes.

[0106] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0107] Step 1:

[0108] The user accesses the system's upload screen using the terminal's web browser, selects the document file (PDF, text file, etc.), and clicks the upload button.

[0109] Input: Document file

[0110] Output: The document file is sent to the server.

[0111] Operation: When the user operates the application, the document file is sent to the server via an HTTP request and temporarily stored in storage.

[0112] Step 2:

[0113] The server analyzes the received data file.

[0114] Input: Uploaded file

[0115] Output: Parsed text data

[0116] Operation: For PDF files, it uses the PDF parser (pdfminer.six) to extract text data. For text files, it reads the content directly. If necessary, it also uses OCR (Tesseract OCR) technology to convert characters in images into text.

[0117] Step 3:

[0118] The server uses natural language processing (NLP) technology (NLTK) to extract keywords and important sentences from the analyzed text data.

[0119] Input: Parsed text data

[0120] Output: Extracted keywords and important sentences

[0121] How it works: NLP technology analyzes text and converts the structure of the material into an easy-to-understand format.

[0122] Step 4:

[0123] Based on the extracted information, the server uses a generative AI model to generate an explanatory scenario.

[0124] Input: Extracted keywords and important sentences

[0125] Output: Explaining scenario

[0126] How it works: A prompt for scenario generation is input into the generative AI model, which then creates a scenario that includes the flow of explanations, narration content, and insertion points for visual materials. For example, the prompt might be, "Please create a scenario that explains the planets in the solar system for use in elementary school science classes."

[0127] Step 5:

[0128] The server creates narration audio based on the generated scenario using a speech synthesis API (Google Text-to-Speech API).

[0129] Input: Explaining scenario

[0130] Output: Narration audio file

[0131] Operation: The scenario text is input into the speech synthesis API, synthetic speech is generated, and saved as a narration audio file.

[0132] Step 6:

[0133] Based on the generated scenario, the server creates visual materials using image generation AI and image editing tools.

[0134] Input: Explaining scenario

[0135] Output: Visual material (image file)

[0136] How it works: Using an image generation AI (e.g., DALL-E) or an image editing tool (Adobe Photoshop), the necessary illustrations and diagrams are generated and saved as image files.

[0137] Step 7:

[0138] The server combines the narration audio and visual material to generate a video.

[0139] Input: Narration audio file, visual material

[0140] Output: Generated video file

[0141] How it works: Use a video editing engine (e.g. FFmpeg) to combine audio and visual elements on the timeline, add transition effects and text overlays to create a video, and save the finished video in an appropriate format, such as MP4.

[0142] Step 8:

[0143] The user can preview the generated animation and provide any necessary corrections.

[0144] Input: Generated video file

[0145] Output: Correction instructions

[0146] How it works: The user plays the video and provides feedback using the preview functionality provided by the server. The server receives correction requests, re-edits the relevant parts, and updates the video.

[0147] Step 9:

[0148] Users can then download the final edited video or generate a shareable link to share it with others.

[0149] Input: Final generated video

[0150] Output: Download link or share link

[0151] How it works: The server generates a secure download link and provides it to the user. The user can then download the video or share it with others using a sharing link. The server also provides the ability to email the sharing link or generate a QR code.

[0152] (Application example 1)

[0153] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0154] Logistics centers need a method to efficiently and quickly communicate work procedures and equipment operation methods to workers. Traditionally, manuals using text and still images have been widely used, but they have problems such as being difficult to understand and being difficult to update. In addition, when new work procedures or equipment are introduced, training is required, which takes time and effort. To solve these problems, a more effective and easy-to-understand method is needed.

[0155] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0156] In this invention, the server includes means for uploading materials, means for analyzing the contents of the uploaded materials, means for generating an explanatory scenario based on the analyzed contents, means for creating a narration voice and visual materials based on the generated scenario, means for generating a video by integrating the narration voice and visual materials, means for customizing the generated video for the logistics center and displaying it on a smart terminal or a wearable device so that workers can check it, means for previewing the generated video and accepting correction instructions, and means for providing the final generated video to the user. This makes it possible to provide work procedures and equipment operation methods in a video format that is instantly easy to understand.

[0157] "Documents" are document data uploaded by users, such as PDFs or text files.

[0158] "Uploading means" refers to a means that has the function of allowing a user to transfer materials to a server via a network.

[0159] "Means for analyzing" means means including technology for analyzing the content of uploaded materials and converting them into an understandable form.

[0160] An "explanatory scenario" is a plan that defines the flow and sequence of video and audio, generated based on the content of the analyzed materials.

[0161] "Narration voice" is voice data created based on a commentary scenario, and conveys the commentary content through sound.

[0162] "Visual materials" are visual materials such as illustrations and diagrams used to complement the explanatory scenario.

[0163] "Means for generating moving images" means means including technology for integrating narration audio and visual materials into a single moving image format.

[0164] "Customizing for the distribution center" means adjusting the format and content of the generated video to be suitable for use in the distribution center.

[0165] "Smart devices" refer to portable information terminals such as smartphones and tablets.

[0166] A "wearable device" is a device that can be worn by a worker and that displays information.

[0167] "Preview" means temporarily viewing the generated video for final confirmation.

[0168] A "correction instruction" is a request for correction or modification made by a user to a generated video.

[0169] The present invention is a system for automatically generating training videos for a logistics center, which operates mainly on a server and a user terminal. A specific embodiment of the system is described below.

[0170] System configuration

[0171] The system consists of the following main components:

[0172] 1. User devices: Including smartphones, PCs, smart devices, wearable devices, etc. Data is uploaded from here.

[0173] 2. Server: The main component that analyzes materials, generates scenarios, synthesizes voice, generates visual materials, and generates videos.

[0174] 3. Network: Responsible for data communication between user terminals and servers.

[0175] Key Features

[0176] Upload materials:

[0177] Materials (PDF or text files) are uploaded to the server from the user's device. Users select and upload materials using a browser-based interface.

[0178] Content analysis of the material:

[0179] The server analyzes the uploaded documents, extracting text data using a PDF parser or text parser, and extracting important keywords and sentences using natural language processing (NLP). It also converts text in images into text using OCR technology, if necessary.

[0180] Generate an explanatory scenario:

[0181] The server generates a commentary scenario based on the analyzed text data, which determines the flow of the video and defines the insertion points for the narration content and necessary visual materials.

[0182] Narration audio and visual material creation:

[0183] The server uses Text-to-Speech (TTS) technology to create narration based on the generated scenario, and generates visual materials (illustrations, diagrams, etc.) using image generation AI and image editing tools.

[0184] Video Generation:

[0185] The server generates a video by integrating the narration audio and visual materials. Using a video editing engine (such as FFmpeg or OpenShot), the materials are placed on a timeline and transition effects and text overlays are added to create a visually appealing video.

[0186] Customize and view the generated video:

[0187] The generated videos are customized for the logistics center and displayed on smart terminals and wearable devices, allowing workers to easily review the videos and apply them to their work.

[0188] Preview and fix:

[0189] The user can preview the generated video and check it in their browser. The server provides a preview function that allows the user to play the video, check the content, and then provide feedback on any necessary corrections.

[0190] Final video credit:

[0191] Users can download the final video or generate a share link to share it with others. The server generates a secure download link, allowing users to save the video locally.

[0192] Specific examples

[0193] For example, if you upload an operating procedure manual (PDF) for a newly introduced packaging machine, the system will analyze the contents and generate a step-by-step scenario like the one below.

[0194] "Step 1: Power on the machine."

[0195] "Step 2: Select the appropriate settings in the panel."

[0196] "Step 3: Place the item in the machine."

[0197] "Step 4: Press the Start button."...and so on.

[0198] The training video is then created by synthesizing the voice and inserting images and diagrams corresponding to each step.

[0199] Prompt Sentence Examples

[0200] Below are prompts for inputting new information into a generative AI model.

[0201] Analyze the following procedure, generate narration text that clearly explains each step, and generate a scenario for creating a training video based on that.

[0202] Instructions:

[0203] Turn on the machine.

[0204] Select the appropriate settings in the panel.

[0205] Place the product in the machine.

[0206] Press the start button.

[0207] Make sure the product is packaged.

[0208] Stop the machine.

[0209] By following the above steps, it is possible to efficiently generate training videos for logistics centers.

[0210] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0211] Step 1:

[0212] This is the process of uploading materials from a user's device. The user uses a browser-based interface to select materials such as PDF or text files and clicks the upload button. The input is the file specified by the user, and the output is the file transfer to the server. This operation sends the selected materials to the server.

[0213] Step 2:

[0214] This is the process of analyzing the content of documents on the server. After the server receives the uploaded documents, it first extracts the text data using a PDF parser or text parser. The input is the uploaded document, and the output is the extracted text data. At the same time, OCR technology is used to extract the text information from the image, thereby extracting pure text information.

[0215] Step 3:

[0216] This is a process that generates an explanatory scenario based on the text data analyzed by the server. The server uses natural language processing (NLP) technology to extract keywords and important sentences from the text and analyze the structure of the material. The input is the extracted text data, and the output is an explanatory scenario. The scenario defines the flow of the video.

[0217] Step 4:

[0218] This is a process that creates narration audio based on a scenario generated on the server. The server uses Text-to-Speech (TTS) technology to convert the scenario text into audio. The input is the explanatory scenario, and the output is the narration audio file. This generates audio data corresponding to the scenario.

[0219] Step 5:

[0220] At the same time, the server creates visual materials. Image generation AI and image editing tools are used to create the necessary illustrations and diagrams based on the explanatory scenario. The input is the explanatory scenario, and the output is the visual materials. This generates image data to visually complement the data.

[0221] Step 6:

[0222] This is the process of integrating narration audio and visual materials on the server to generate a video. Using a video editing engine (such as FFmpeg or OpenShot), the materials are placed on a timeline and transition effects and text overlays are added. The input is the narration audio and visual materials, and the output is the generated video file.

[0223] Step 7:

[0224] This process customizes the generated video for the logistics center and displays it on a smart terminal or wearable device so that workers can review it. The input is the generated video file, and the output is the video displayed on a smart terminal or wearable device. This allows workers to learn their work while watching the video on-site.

[0225] Step 8:

[0226] This process previews the generated video and accepts correction instructions. The user plays the video in a browser and provides feedback on any necessary corrections. The input is the generated video file, and the output is correction instructions from the user.

[0227] Step 9:

[0228] This is the process of providing the final generated video to the user. The server generates a secure download link so that the user can save the video locally, and also generates a sharing link so that the video can be shared with other people. The input is the generated video file, and the output is the download link or sharing link.

[0229] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0230] This invention is a system that uploads materials, analyzes their contents, generates videos, and provides them in an optimal format while recognizing the user's emotions. The specific program processing flow and system operation are described in detail below.

[0231] Uploading and Loading Materials

[0232] The user uploads the selected material (PDF, text file, etc.) to the system. They open a web browser on their device, select the material on the system's upload screen, and click the upload button. This operation sends the material to the server. Furthermore, the emotion engine recognizes the user's emotional state (happiness, sadness, surprise, etc.) during uploading using a facial recognition system or voice input.

[0233] Text analysis of materials

[0234] The server receives and saves the uploaded file. It checks the file format and prepares to analyze the content of the document using a PDF parser or text parser. The content of the document is then analyzed. In the case of a PDF, the PDF parser is used to extract text data, and OCR technology is used to convert the characters in the image into text. In the case of a text file, the text data is read directly. Natural language processing (NLP) technology is then used to extract key information (keywords, important sentences, paragraphs, etc.). Based on the analyzed text data, the emotion engine adjusts the content output according to the user's emotional state (e.g., joy, sadness, surprise, etc.).

[0235] Scenario generation based on the content of the materials

[0236] The server generates a scenario for the commentary based on the analyzed text data. This scenario includes the flow of the commentary, narration content, key points, and insertion points for visual materials. Furthermore, an emotion engine evaluates the user's emotions and adjusts the content of the scenario accordingly. For example, if the user is having fun, the tone will be changed to a slightly more casual one, and if the user is sad, the explanation will be given in a warmer tone.

[0237] Voice synthesis and visual material creation

[0238] The server creates narration audio and visual materials based on the generated scenario. Text-to-speech (TTS) technology is used to convert text into audio and create the narration. At the same time, visual materials (illustrations, diagrams, etc.) are generated using image generation AI and image editing tools. The emotion engine selects the audio tone and visual style (e.g., softer colors and designs) that match the user's emotional state.

[0239] Video Generation

[0240] The server generates a video by integrating the narration audio and visual materials. It then uses a video editing engine (e.g., FFmpeg or OpenShot) to place the audio files and visual materials on a timeline. It also adds transition effects and text overlays to improve the visual quality of the video. Based on instructions from the emotion engine, it also adds effects and music that match the user's emotions.

[0241] Preview and edit your video

[0242] The server provides the generated video for the user to preview. The user then checks the video on their browser to see if the content matches their intentions. At this time, the emotion engine recognizes the user's emotions again and evaluates their reaction to the video content. If necessary, the user can specify areas to be corrected and provide feedback. The re-edited parts are adjusted taking into account the user's emotional state.

[0243] Download or share the video

[0244] The server delivers the final video that the user approves. The user can then download the video or generate a sharing link to share it with others. The server generates a secure download link so the user can save the video locally. It also generates a viewing link for internal or external sharing, and provides email and QR code generation functions as needed.

[0245] As a concrete example, consider the case where a parent creates an instructional video for their child about "constellations in space." When the parent uploads the material, the emotion engine recognizes the parent's emotions and detects that the parent is having fun. The server creates an instructional scenario with a fun, casual tone and selects bright colors and friendly visual materials. The parent previews the generated video and makes fine adjustments based on the parent's emotional response. After the corrections are reflected, the parent can download the final video and show it to their child.

[0246] This system allows users to quickly convey accurate and easy-to-understand information in a video format that best suits their emotional state. Taking emotions into consideration leads to more effective communication.

[0247] The processing flow will be explained below.

[0248] Step 1:

[0249] The user uploads the selected material (PDF, text file, etc.) to the system. They open a web browser on their device, select the material on the system's upload screen, and click the upload button. At this time, the emotion engine recognizes the user's emotional state from their facial expressions and voice.

[0250] Step 2:

[0251] The server receives and stores the uploaded file, checks the file format, and prepares the content for analysis using a PDF parser or text parser.

[0252] Step 3:

[0253] The server analyzes the content of the document. In the case of a PDF, it uses a PDF parser to extract text data and OCR technology to convert the characters in the image into text. In the case of a text file, it reads the text data directly. Next, it uses natural language processing (NLP) technology to extract key information (keywords, important sentences, paragraphs, etc.).

[0254] Step 4:

[0255] The server generates explanatory scenarios based on the analyzed text data and the user's emotional state. For example, if the user looks happy, it generates a scenario with a casual and cheerful tone, and if the user looks sad, it generates a scenario with a gentler and warmer tone.

[0256] Step 5:

[0257] The server creates narration audio and visual materials based on the scenario. It uses text-to-speech (TTS) technology to convert the scenario text into narration audio. At the same time, it follows instructions from the emotion engine to create visual materials with designs and colors that match the user's emotional state. For example, bright colors are used for cheerful emotions, and softer colors are used for calm emotions.

[0258] Step 6:

[0259] The server generates a video by integrating the narration audio and visual materials. A video editing engine (e.g., FFmpeg or OpenShot) is used to place the audio files and visual materials on a timeline. Furthermore, transition effects and text overlays are added to improve the visual quality of the video. The emotion engine also selects effects and background music that match the user's emotions.

[0260] Step 7:

[0261] The server provides the generated video for the user to preview. The user can then view the video in their browser and check whether the content matches their intent. At this time, the emotion engine again recognizes the user's emotions and evaluates their reaction to the video.

[0262] Step 8:

[0263] The user provides feedback on the parts they want to edit, and the server receives the instructions. The emotion engine re-evaluates the user's emotional state, makes any necessary adjustments, and re-edits the video. The re-edited video is then provided for preview again, and the user makes a final confirmation.

[0264] Step 9:

[0265] The server delivers the final video that the user approves. The user can then download the video or generate a sharing link to share it with others. The server generates a secure download link so the user can save the video locally. It also generates a viewing link for internal or external sharing, and provides email and QR code generation functions as needed.

[0266] This series of steps allows users to easily create and quickly deliver accurate, easy-to-understand explainer videos that best suit their emotional state, making information delivery more effective and personalized.

[0267] Example 2

[0268] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0269] In recent years, the demand for digital content has increased, and there is a particular need for the generation of video content that is optimized for individual users' emotions. However, conventional systems have had difficulty providing adaptive video content that takes into account user emotions throughout the entire process: uploading materials, analyzing content, generating scenarios, creating narration audio, creating visual materials, generating videos, and previewing and editing. This has resulted in a poor user experience and made it difficult to achieve effective communication.

[0270] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0271] In this invention, the server includes means for uploading materials, means for analyzing the content of the uploaded materials, means for generating an explanatory scenario based on the analyzed content, means for creating a narration voice and visual materials based on the generated scenario, means for generating a video by integrating the narration voice and visual materials, means for previewing the generated video and accepting correction instructions, means for providing the final generated video to a user, means for recognizing the user's emotions using a face recognition system or voice input when uploading and previewing the materials, and means for adjusting the content of the explanatory scenario and visual materials based on the user's emotions, thereby enabling the rapid and effective creation of video content optimized for the user's emotional state.

[0272] "Means for uploading materials" refers to the technical means that provides an interface for users to send materials they possess to the system and store them on the server.

[0273] "Means for analyzing the content of uploaded materials" refers to the technical means by which the server extracts and understands the content of the materials received by the server using text analysis and image processing technology.

[0274] The "means for generating an explanatory scenario" is a technical means for automatically creating a scenario including explanations, narration flow, important points, etc. based on the analyzed content.

[0275] "Means for creating narration audio and visual materials" refers to the technical means for generating narration audio and visual materials using voice synthesis technology and image generation technology based on the generated scenario.

[0276] "Means for generating a video by integrating narration audio and visual materials" refers to means for combining narration audio and visual materials into a single video file, and including video editing technology for improving editing quality.

[0277] The "means for previewing the video and accepting correction instructions" refers to a technical means for providing the generated video so that the user can check it, and accepting correction instructions from the user as necessary.

[0278] "Means for providing the final video to the user" refers to the technical means for providing the final video approved by the user in a downloadable format or as a shareable link.

[0279] A "facial recognition system" is a technology that analyzes a user's facial image and identifies a specific emotional state.

[0280] "Voice input" is a technology that analyzes a user's voice, converts the content into text, and identifies the user's emotional state from the voice.

[0281] "Means for adjusting the content of explanatory scenarios and visual materials based on the user's emotions" refers to technical means for adaptively adjusting the tone and style of the scenarios and visual materials to be generated using the user's emotional data.

[0282] This invention is a system that uploads materials, analyzes their contents, generates videos, and provides them in the most appropriate format while recognizing the user's emotions. The program processing flow and operation of this system are described in detail below.

[0283] Uploading and Loading Materials

[0284] The user uploads the selected material (PDF, text file, etc.) to the system. They open a web browser on their device, select the material on the system's upload screen, and click the "Upload" button. This operation sends the material to the server. Furthermore, the emotion engine recognizes the user's emotional state (happiness, sadness, surprise, etc.) during uploading using a facial recognition system or voice input.

[0285] Text analysis of materials

[0286] The server receives and saves the uploaded file. It checks the file format and prepares the content for analysis using a PDF parser or text parser. The content of the document is then analyzed. For PDFs, a PDF parser (e.g., PyMuPDF or PDFMiner) is used to extract text data, and OCR technology (e.g., Tesseract OCR) is used to convert the characters in the image into text. For text files, the text data is read directly. Next, natural language processing (NLP) technology (e.g., spaCy or NLTK) is used to extract key information (keywords, important sentences, clauses, etc.). Based on the analyzed text data, the emotion engine adjusts the content output according to the user's emotional state (e.g., joy, sadness, surprise, etc.).

[0287] Scenario generation based on the content of the materials

[0288] The server generates a scenario for the commentary based on the analyzed text data. This scenario includes the flow of the commentary, narration content, key points, and insertion points for visual materials. Furthermore, an emotion engine evaluates the user's emotions and adjusts the content of the scenario accordingly. For example, if the user is having fun, the tone will be changed to a slightly more casual one, and if the user is sad, the explanation will be given in a warmer tone.

[0289] Voice synthesis and visual material creation

[0290] The server creates narration audio and visual materials based on the generated scenario. It uses text-to-speech (TTS) technology (e.g., Google Text-to-Speech or Amazon Polly) to convert text into speech and create the narration. At the same time, visual materials (e.g., illustrations, diagrams, etc.) are generated using image generation AI (e.g., DALL-E or MidJourney) or image editing tools (e.g., Photoshop). The emotion engine selects the voice tone and visual style (e.g., softer colors and designs) that match the user's emotional state.

[0291] Video Generation

[0292] The server generates a video by integrating the narration audio and visual materials. It uses a video editing engine (e.g., FFmpeg or OpenShot) to place the audio files and visual materials on a timeline. It also adds transition effects and text overlays to improve the visual quality of the video. Based on instructions from the emotion engine, it also adds effects and music that match the user's emotions.

[0293] Preview and edit your video

[0294] The server provides the generated video for the user to preview. The user then checks the video on their browser to see if the content matches their intentions. At this time, the emotion engine recognizes the user's emotions again and evaluates their reaction to the video content. If necessary, the user can specify areas to be corrected and provide feedback. The re-edited parts are adjusted taking into account the user's emotional state.

[0295] Download or share the video

[0296] The server delivers the final video that the user approves. The user can then download the video or generate a sharing link to share it with others. The server generates a secure download link so the user can save the video locally. It also generates a viewing link for internal or external sharing, and provides email and QR code generation functions as needed.

[0297] Examples of concrete examples and prompts

[0298] For example, consider a case where a parent creates an instructional video for their child about "constellations in space." When the parent uploads the material, the emotion engine recognizes the parent's emotions and detects that the parent is having fun. The server creates an instructional scenario with a fun, casual tone and selects bright colors and friendly visual materials. The parent can preview the generated video and make fine adjustments based on the parent's emotional response. After the corrections are reflected, the parent can download the final video and show it to their child.

[0299] Examples of prompts include:

[0300] "How can I upload a PDF file, detect the emotion of joy, and generate a casual explainer video based on that?"

[0301] "Please explain the steps for users to upload materials online, perform real-time emotion recognition, and use that data to create a video."

[0302] This system allows users to quickly convey accurate and easy-to-understand information in a video format that best suits their emotional state. Taking emotions into consideration leads to more effective communication.

[0303] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0304] Step 1:

[0305] The user uploads the material.

[0306] Input: User selected material (PDF, text file, etc.).

[0307] Operation: Open the device's web browser, select the file on the system's upload screen, and click the "Upload" button. Emotional data is also acquired through the user's facial recognition system and voice input.

[0308] Output: The selected material file is sent to the server, along with the user's emotion data.

[0309] Step 2:

[0310] The server receives the material.

[0311] Input: Uploaded data files, user emotion data.

[0312] Operation: The server saves the received file in the specified directory. It determines the file format (PDF, text file, etc.).

[0313] Output: The path of the saved document file, file format information.

[0314] Step 3:

[0315] The server analyzes the data.

[0316] Input: Saved material file, file format information.

[0317] How it works: For PDF files, the server extracts text using a PDF parser (e.g., PyMuPDF or PDFMiner) and converts characters in images to text using OCR technology (e.g., Tesseract OCR). For text files, the server reads the text data directly and extracts key information using natural language processing (NLP) technology (e.g., spaCy or NLTK).

[0318] Output: Parsed text data, extracted key information.

[0319] Step 4:

[0320] The emotion engine evaluates the user's emotions.

[0321] Input: User emotion data, parsed text data, extracted key information.

[0322] How it works: The emotion engine evaluates the analyzed text data based on the user's emotional state (happiness, sadness, surprise, etc.) and adjusts the content output.

[0323] Output: Emotion evaluation results, adjusted content output instructions.

[0324] Step 5:

[0325] The server generates the scenario.

[0326] Input: Parsed text data, extracted key information, sentiment evaluation results, and tailored content output instructions.

[0327] Operation: Based on these input data, the server generates a commentary scenario, which includes the flow of the commentary, narration content, important points, and insertion points for visual materials.

[0328] Output: The generated explanatory scenario.

[0329] Step 6:

[0330] The server creates the audio and visual material.

[0331] Input: Generated commentary scenario, emotion evaluation results.

[0332] How it works: The server uses voice synthesis technology (e.g., Google Text-to-Speech or Amazon Polly) to create the narration, and image generation AI (e.g., DALL-E or MidJourney) or image editing tools (e.g., Photoshop) to generate the visuals. The voice tone and visual style are adjusted to match the user's emotional state.

[0333] Output: Generated narration audio and visual materials.

[0334] Step 7:

[0335] The server generates the video.

[0336] Input: Generated narration audio, visual materials, and emotion evaluation results.

[0337] How it works: The server uses a video editing engine (e.g., FFmpeg or OpenShot) to combine audio and visual material into a single video file. It also adds transition effects and text overlays to enhance the visual quality. It also adds effects and music that suit the user's emotions.

[0338] Output: The generated video file.

[0339] Step 8:

[0340] The server provides a preview of the generated video.

[0341] Input: Generated video files, emotion evaluation results.

[0342] How it works: The server provides the user with a link to preview the video in their browser. The user reviews the video content and makes corrections as needed. The emotion engine again recognizes the user's emotions and evaluates their reactions during the preview.

[0343] Output: User's correction instructions, updated emotion evaluation results.

[0344] Step 9:

[0345] The server provides the final video.

[0346] Input: Generated video file, correction instructions from user.

[0347] How it works: The server re-edits the video based on the correction instructions and generates the final version. It then provides the final video to the user and generates a download link and a sharing link.

[0348] Output: Final video file, secure download link, and share link.

[0349] (Application example 2)

[0350] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0351] In conventional video generation systems, the process of analyzing the content of materials and creating videos is uniformly created without considering the user's emotional state, making it difficult to provide personalized content. Furthermore, in order for users to optimally understand and enjoy the content of videos, adjustments based on the user's emotions are necessary, but there has been a lack of systems that can achieve this.

[0352] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0353] In this invention, the server includes means for uploading materials, means for analyzing the content of the uploaded materials, means for generating an explanatory scenario based on the analyzed content, means for creating a narration voice and visual materials based on the generated scenario, means for generating a video by integrating the narration voice and visual materials, means for previewing the generated video and accepting correction instructions, means for recognizing the emotional state of the user and adjusting the scenario and visual style according to the emotion, and means for providing the user with the final generated video, thereby making it possible to provide optimal content according to the user's emotion.

[0354] "Means for uploading materials" refers to a device or method for uploading materials owned by a user into the system.

[0355] "Means for analyzing the content of uploaded material" means a device or method used to computerize and understand the content of uploaded material.

[0356] The "means for generating a commentary scenario" is a device or method for creating a plot or text for explanation or narration based on the analyzed content.

[0357] The "means for creating narration audio and visual materials" refers to a device or method for creating audio data and visual materials based on a generated scenario.

[0358] "Means for generating video" refers to a device or method for combining narration audio and visual materials to edit and generate a single video.

[0359] The "means for previewing the generated video and accepting correction instructions" refers to a device or method that allows a user to view the generated video in advance and indicate any necessary corrections.

[0360] "Means for recognizing the user's emotional state and adjusting the scenario and visual style according to that emotion" refers to a device or method that reads the user's emotions from their facial expressions, voice, etc., and changes or adjusts the content of the scenario and the visual design based on that.

[0361] The "means for providing the final video to the user" refers to a device or method that allows the user to download the completed video or share it with others.

[0362] An embodiment of the present invention will be described. This system analyzes materials uploaded by users and generates optimal explanatory videos while recognizing their emotional state. This system makes it possible to provide personalized video content that reflects the emotions of each individual user.

[0363] Uploading and Loading Materials

[0364] Users access the system's upload screen using a smartphone or computer browser. On the upload screen, users select and upload documents in PDF or text file format. At the same time as uploading, the system uses the device's built-in camera and microphone to recognize the user's emotional state through a facial recognition system or voice input. This emotion recognition uses facial expression recognition software and voice analysis software.

[0365] Text analysis of materials

[0366] The server receives the uploaded documents and extracts the text data using a PDF parser and OCR technology. This converts the text information from the images in the documents into text. Key information is then extracted using natural language processing (NLP). At this stage, the server also adjusts the analysis results based on the user's emotional state.

[0367] Scenario Generation

[0368] The server generates a commentary scenario for the user based on the emotional state and the analyzed text data. The generated scenario includes the flow of the commentary, the narration content, and the insertion points of visual materials. The emotion engine evaluates the user's emotions and adjusts the content of the scenario. For example, if the user is having fun, it will use casual language and select visually bright materials.

[0369] Creation of narration audio and visual materials

[0370] The server uses text-to-speech (TTS) technology to generate narration based on the generated scenario. At the same time, it creates visual materials such as illustrations and diagrams using generative AI models and image editing tools. At this stage, it also selects the style and tone that best suits the user's emotions.

[0371] Video Generation

[0372] The server combines the generated narration audio with the visual material and generates a video using a video editing engine (e.g., FFmpeg or OpenShot). The narration and visual material are placed on a timeline, and the visual quality is improved by adding transition effects and text overlays. Again, visual effects and music are applied based on the instructions of the emotion engine.

[0373] Preview and edit your video

[0374] The video generated by the server is previewed by the user. The user checks the video in their browser to ensure that the content is as intended. At the same time, the emotion engine evaluates the user's emotional response to the video. If necessary, the user can specify corrections and the video is re-edited.

[0375] Download or share the video

[0376] The final video is then delivered to the user via the server, who can either download the video or generate a shareable link to share it with others. The server generates a secure download link and also provides a viewing link for internal or external sharing.

[0377] Specific examples

[0378] For example, if a parent wants to create a learning video for their child on "Earth Geography," the system will automatically detect the parent's emotional state when they upload the materials. If a happy emotion is detected, a casual and fun-toned commentary scenario will be generated. Brightly colored diagrams and animations will be selected, and the parent can fine-tune the video while previewing it. Finally, a fun-toned geography learning video will be generated, which the parent can show to their child.

[0379] Prompt Sentence Examples

[0380] "Analyze this material and generate fun narration and colorful visuals to create a geography learning video for children."

[0381] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0382] Step 1:

[0383] Users access the system's upload screen using their devices, select and upload documents in PDF or text file format. The input data is the document file owned by the user, and the output data is the document file sent to the server. The user's emotional state is also collected at the same time using a facial recognition system and voice input. The input data is the user's facial expressions and voice, and the output data is the emotion recognition results.

[0384] Step 2:

[0385] The server analyzes the received document file. In the case of PDF, it uses a PDF parser to extract the text data and OCR technology to convert the characters in the image into text. The input data is the document file sent to the server, and the output data is the extracted text data.

[0386] Step 3:

[0387] The server uses natural language processing (NLP) techniques to extract keywords and important sentences from the extracted text data. The input data is the extracted text data, and the output data is a list of keywords and important sentences.

[0388] Step 4:

[0389] The server generates an explanatory scenario based on the emotion recognition results and the analyzed text data. The input data is a list of keywords and important sentences and the emotion recognition results, and the output data is the generated explanatory scenario. The emotion recognition results affect the tone and content of the scenario. For example, a casual scenario is generated for a positive emotional state.

[0390] Step 5:

[0391] The server uses text-to-speech (TTS) technology to create a narration voice based on the generated scenario. The input data is the generated commentary scenario, and the output data is a narration voice file. The TTS engine converts the text into speech.

[0392] Step 6:

[0393] The server creates visual materials using generative AI models and image editing tools. The input data is the generated explanatory scenario, and the output data is a set of visual materials. Diagrams and illustrations are generated to match the key points of the scenario.

[0394] Step 7:

[0395] The server uses a video editing engine (e.g., FFmpeg or OpenShot) to combine the narration audio and visual materials to generate a video. The input data is a narration audio file and a set of visual materials, and the output data is the generated video file. Transition effects and text overlays are also added.

[0396] Step 8:

[0397] The server allows the user to preview the generated video. The user checks the video on the browser and provides feedback on any corrections. The input data is the generated video file, and the output data is the user's feedback. The emotion engine reevaluates the user's emotions about the video and identifies areas that need to be re-edited.

[0398] Step 9:

[0399] The server re-edits the video based on the user's feedback and makes final adjustments according to the emotions. The input data are the user's feedback and the emotion re-evaluation results, and the output data is the final video file.

[0400] Step 10:

[0401] The server provides the final video to the user, who can then download it or share it with others via a sharing link. The input data is the final video file, and the output data is a download link and a sharing link. The server generates a secure download link and provides a viewing link.

[0402] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0403] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0404] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0405] [Second embodiment]

[0406] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0407] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0408] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0409] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0410] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0411] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0412] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0413] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0414] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0415] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0416] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0417] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0418] This invention is a system that uploads materials, analyzes the contents, generates videos, and provides them to users. The specific program processing flow and system operation are described in detail below.

[0419] Uploading and Loading Materials

[0420] The user uploads the selected book, dictionary, picture book, or material (PDF, text file, etc.) to the system. Using the device's web browser, the user selects the material on the designated upload screen and clicks the upload button. This operation sends the material to the server.

[0421] Text analysis of materials

[0422] The server analyzes the uploaded document. It uses a PDF parser and text parser to extract the text data from the document. It then uses natural language processing (NLP) technology to extract keywords and important sentences from the text and analyze the structure of the document. If necessary, OCR (optical character recognition) technology is used to convert text in images into text.

[0423] Scenario generation based on the content of the materials

[0424] The server generates a commentary scenario based on the analyzed text data. This is the process of determining the flow of the video and defining the narration content and insertion points for necessary visual materials. Structured data is created, specifying the commentary order and detailed narration content for each section.

[0425] Voice synthesis and visual material creation

[0426] The server creates narration audio and visual materials based on the generated scenario. It uses Text-to-Speech (TTS) technology to convert text into audio and create the narration. At the same time, visual materials (illustrations, diagrams, etc.) are generated using image generation AI and image editing tools.

[0427] Video Generation

[0428] The server combines the narration audio with the visual material to generate the video. Using a video editing engine (such as FFmpeg or OpenShot), the material is placed on a timeline and transition effects and text overlays are added to create a visually appealing video. At this stage, the video is exported to its final form and saved in an appropriate format, such as MP4.

[0429] Preview and edit your video

[0430] The user can preview the generated video and check it on the browser. The server provides a preview function that enables this, and after the user plays the video and checks the content, they can provide feedback on any necessary corrections. Upon receiving the correction request, the server re-edits the relevant parts and updates the video.

[0431] Download or share the video

[0432] Users can download the final video or generate a sharing link to share it with others. The server generates a secure download link, allows users to save the video locally, and generates a viewing link for internal or external sharing, and also provides the ability to send an email or generate a QR code if desired.

[0433] As a concrete example, let's consider the case where a parent creates an explanatory video for their child about the planets of the solar system. The parent uploads a PDF of an elementary school science textbook and inputs it into the system. The server analyzes the material and extracts information about the planets in the solar system. An explanatory scenario is then generated, and narration audio and planet illustrations are created. These materials are then integrated into a video. The parent can use the preview function to check the video and make any final corrections. After the corrections are complete, the parent can download the final video and show it to their child.

[0434] Through this series of steps, users can easily provide accurate information in video format, saving a great deal of time and effort. By using this system, information can be transmitted quickly and effectively.

[0435] The processing flow will be explained below.

[0436] Step 1:

[0437] The user uploads the selected material (PDF, text file, etc.) to the system by opening a web browser on the terminal, selecting the material on the system's upload screen, and clicking the upload button.

[0438] Step 2:

[0439] The server receives and stores the uploaded file, first checking the file format and preparing to parse the content using a PDF parser or text parser.

[0440] Step 3:

[0441] The server analyzes the content of the document. In the case of a PDF, it uses a PDF parser to extract the text data and OCR technology to convert the characters in the image into text. In the case of a text file, it reads the text data directly. Next, it uses natural language processing (NLP) technology to extract key information (keywords, important sentences, paragraphs, etc.).

[0442] Step 4:

[0443] The server generates a commentary scenario based on the analyzed text data. The scenario includes the flow of the commentary, narration content, important points, and points where visual materials should be inserted. The scenario is organized as structured data.

[0444] Step 5:

[0445] The server creates narration audio and visual materials based on the scenario. It uses text-to-speech (TTS) technology to convert the scenario text into narration audio. At the same time, it uses image generation AI and image editing tools to generate the necessary visual materials, such as illustrations and diagrams.

[0446] Step 6:

[0447] The server generates a video by integrating the narration audio and visual materials. Using a video editing engine (e.g., FFmpeg or OpenShot), the audio files and visual materials are placed on a timeline. Furthermore, transition effects and text overlays are added to improve the visual quality of the video.

[0448] Step 7:

[0449] The server provides the generated video so that the user can preview it. The user can check the video in the browser and check whether the content matches the user's intention. If necessary, the user can specify corrections and provide feedback.

[0450] Step 8:

[0451] The server receives correction instructions from the user, re-edits the relevant parts, and provides the re-edited video for preview again so that the user can make a final check.

[0452] Step 9:

[0453] The server then provides the final video that the user has approved. The user can then download the video or generate a sharing link to share it with others. The server generates secure download links and viewing links, and also provides email and QR code generation functions as needed.

[0454] By following the above steps, users can easily create and quickly provide accurate and easy-to-understand instructional videos based on materials of their own choosing. This system can be used for a wide range of purposes, such as home education and creating business manuals in companies.

[0455] Example 1

[0456] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0457] Conventional document analysis systems often lack sufficient methods for efficiently understanding the content of documents. In particular, it is difficult to quickly and accurately analyze vast amounts of data and provide the information in a format that is easy for users to understand, which requires time and effort. Furthermore, simply providing the content of documents in text format makes it difficult to convert it into a visually understandable format, making it ineffective for use in education, presentations, and other applications. Therefore, there is a growing demand for a system that can analyze uploaded documents and provide information in video format based on the results.

[0458] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0459] In this invention, the server includes means for uploading materials, means for analyzing the content of the uploaded materials, means for generating an explanatory scenario based on the analyzed content, means for creating a narration voice and visual materials based on the generated scenario, means for generating a video by integrating the narration voice and visual materials, means for previewing the generated video and accepting correction instructions, and means for providing the final generated video to the user. This makes it possible to analyze the content of the uploaded materials and provide the information quickly and efficiently in a video format that is easy for the user to understand.

[0460] The "means for uploading materials" is a function for transmitting materials selected by the user to the system and storing them on the server.

[0461] "Means for analyzing the content of uploaded materials" refers to a function for extracting text data and other information from uploaded materials and converting that information into an easily understandable format.

[0462] The "means for generating an explanatory scenario" is a function that creates a scenario for determining the structure of the explanatory video and the content of the narration based on the content of the analyzed materials.

[0463] "Means for creating narration audio and visual materials" refers to a function for creating narration audio and visual materials using voice synthesis technology and image generation technology in accordance with the generated scenario.

[0464] "Means for integrating narration audio and visual materials to generate video" refers to a function that combines the created audio and visual materials on a timeline and outputs them as a completed video.

[0465] The "means for previewing the generated video and accepting correction instructions" is a function that allows the user to check the generated video and provide feedback on parts that need correction.

[0466] The "means for providing the final video to the user" is a function that allows the user to download the finalized video or generate a link that allows the user to share it with other people.

[0467] "Natural language processing technology" is a method for analyzing text data, extracting keywords and important sentences from it, and understanding the content of the material.

[0468] "Speech synthesis technology" is a technology that generates synthetic speech based on text data.

[0469] "Image generation technology" refers to the technology of creating visual materials based on text or other data.

[0470] A "video editing engine" is software or a system for generating and editing videos by combining narration audio and visual materials.

[0471] The present invention provides a system for uploading materials, analyzing the contents of the materials, generating videos, and providing the videos to users. Specific embodiments will be described in detail below.

[0472] Uploading and Loading Materials

[0473] The user accesses the system's upload screen using a web browser on their device and uploads the selected materials (PDF, text files, etc.). The user clicks the upload button, which sends the materials to the server.

[0474] Analysis of data

[0475] The server receives the uploaded documents and temporarily stores them in storage. It then uses a PDF parser (pdfminer.six) to extract text data from the PDF file. For text files, it directly reads the content. It then uses natural language processing (NLP) technology (NLTK) to extract keywords and important sentences from the text and analyze the structure of the document. If necessary, it also converts text within images into text using OCR (optical character recognition) technology (Tesseract OCR).

[0476] Generate a scenario

[0477] The server generates an explanatory scenario based on the analyzed text data. This scenario includes the flow of the video, the narration content, and insertion points for visual materials. To generate the scenario, a generative AI model is used and a prompt is entered. For example, the prompt might read, "Please create a scenario that explains the planets in the solar system for elementary school science classes. I have uploaded a PDF of a science textbook as a reference. Based on this reference, please generate a scenario that includes the characteristics, relative positions, and important information about each planet."

[0478] Voice synthesis and visual material creation

[0479] Based on the generated scenario, the server uses text-to-speech (TTS) technology (Google Text-to-Speech API) to convert text into speech and create the narration. At the same time, it uses image generation AI (e.g., DALL-E) and image editing tools (Adobe Photoshop) to create the necessary visual materials.

[0480] Video generation

[0481] The server integrates the created narration audio with the visual material, and then uses a video editing engine (e.g., FFmpeg) to place the material on a timeline, adding transition effects and text overlays to create a visually appealing video.

[0482] Preview and edit your video

[0483] The user can preview the generated video on the browser and check the content. The server provides a preview function and gives feedback to the user on any corrections that need to be made. The server receives the correction request, re-edits the relevant parts, and updates the video.

[0484] Video provision

[0485] Users can then download the final video or share it with others. The server generates a secure download link and provides it to the user. It can also generate a sharing link for viewing, which can be sent via email or generated as a QR code.

[0486] As a specific example, if a parent wants to create an explanatory video for their child about the planets in the solar system, they upload a PDF of a science textbook to the system. The server analyzes the material and extracts information about the planets in the solar system. The generative AI model then generates a scenario and uses speech synthesis and image generation technologies to create narration audio and illustrations of the planets. These are then integrated to generate a video, and the user (parent) can use the preview function to check the content, make any necessary corrections, and then download or share the final video.

[0487] This system allows users to easily create high-quality videos for educational and presentation purposes.

[0488] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0489] Step 1:

[0490] The user accesses the system's upload screen using the terminal's web browser, selects the document file (PDF, text file, etc.), and clicks the upload button.

[0491] Input: Document file

[0492] Output: The document file is sent to the server.

[0493] Operation: When the user operates the application, the document file is sent to the server via an HTTP request and temporarily stored in storage.

[0494] Step 2:

[0495] The server analyzes the received data file.

[0496] Input: Uploaded file

[0497] Output: Parsed text data

[0498] Operation: For PDF files, it uses the PDF parser (pdfminer.six) to extract text data. For text files, it reads the content directly. If necessary, it also uses OCR (Tesseract OCR) technology to convert characters in images into text.

[0499] Step 3:

[0500] The server uses natural language processing (NLP) technology (NLTK) to extract keywords and important sentences from the analyzed text data.

[0501] Input: Parsed text data

[0502] Output: Extracted keywords and important sentences

[0503] How it works: NLP technology analyzes text and converts the structure of the material into an easy-to-understand format.

[0504] Step 4:

[0505] Based on the extracted information, the server uses a generative AI model to generate an explanatory scenario.

[0506] Input: Extracted keywords and important sentences

[0507] Output: Explaining scenario

[0508] How it works: A prompt for scenario generation is input into the generative AI model, which then creates a scenario that includes the flow of explanations, narration content, and insertion points for visual materials. For example, the prompt might be, "Please create a scenario that explains the planets in the solar system for use in elementary school science classes."

[0509] Step 5:

[0510] The server creates narration audio based on the generated scenario using a speech synthesis API (Google Text-to-Speech API).

[0511] Input: Explaining scenario

[0512] Output: Narration audio file

[0513] Operation: The scenario text is input into the speech synthesis API, synthetic speech is generated, and saved as a narration audio file.

[0514] Step 6:

[0515] Based on the generated scenario, the server creates visual materials using image generation AI and image editing tools.

[0516] Input: Explaining scenario

[0517] Output: Visual material (image file)

[0518] How it works: Using an image generation AI (e.g., DALL-E) or an image editing tool (Adobe Photoshop), the necessary illustrations and diagrams are generated and saved as image files.

[0519] Step 7:

[0520] The server combines the narration audio and visual material to generate a video.

[0521] Input: Narration audio file, visual material

[0522] Output: Generated video file

[0523] How it works: Use a video editing engine (e.g. FFmpeg) to combine audio and visual elements on the timeline, add transition effects and text overlays to create a video, and save the finished video in an appropriate format, such as MP4.

[0524] Step 8:

[0525] The user can preview the generated animation and provide any necessary corrections.

[0526] Input: Generated video file

[0527] Output: Correction instructions

[0528] How it works: The user plays the video and provides feedback using the preview functionality provided by the server. The server receives correction requests, re-edits the relevant parts, and updates the video.

[0529] Step 9:

[0530] Users can then download the final edited video or generate a shareable link to share it with others.

[0531] Input: Final generated video

[0532] Output: Download link or share link

[0533] How it works: The server generates a secure download link and provides it to the user. The user can then download the video or share it with others using a sharing link. The server also provides the ability to email the sharing link or generate a QR code.

[0534] (Application example 1)

[0535] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0536] Logistics centers need a method to efficiently and quickly communicate work procedures and equipment operation methods to workers. Traditionally, manuals using text and still images have been widely used, but they have problems such as being difficult to understand and being difficult to update. In addition, when new work procedures or equipment are introduced, training is required, which takes time and effort. To solve these problems, a more effective and easy-to-understand method is needed.

[0537] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0538] In this invention, the server includes means for uploading materials, means for analyzing the contents of the uploaded materials, means for generating an explanatory scenario based on the analyzed contents, means for creating a narration voice and visual materials based on the generated scenario, means for generating a video by integrating the narration voice and visual materials, means for customizing the generated video for the logistics center and displaying it on a smart terminal or a wearable device so that workers can check it, means for previewing the generated video and accepting correction instructions, and means for providing the final generated video to the user. This makes it possible to provide work procedures and equipment operation methods in a video format that is instantly easy to understand.

[0539] "Documents" are document data uploaded by users, such as PDFs or text files.

[0540] "Uploading means" refers to a means that has the function of allowing a user to transfer materials to a server via a network.

[0541] "Means for analyzing" means means including technology for analyzing the content of uploaded materials and converting them into an understandable form.

[0542] An "explanatory scenario" is a plan that defines the flow and sequence of video and audio, generated based on the content of the analyzed materials.

[0543] "Narration voice" is voice data created based on a commentary scenario, and conveys the commentary content through sound.

[0544] "Visual materials" are visual materials such as illustrations and diagrams used to complement the explanatory scenario.

[0545] "Means for generating moving images" means means including technology for integrating narration audio and visual materials into a single moving image format.

[0546] "Customizing for the distribution center" means adjusting the format and content of the generated video to be suitable for use in the distribution center.

[0547] "Smart devices" refer to portable information terminals such as smartphones and tablets.

[0548] A "wearable device" is a device that can be worn by a worker and that displays information.

[0549] "Preview" means temporarily viewing the generated video for final confirmation.

[0550] A "correction instruction" is a request for correction or modification made by a user to a generated video.

[0551] The present invention is a system for automatically generating training videos for a logistics center, which operates mainly on a server and a user terminal. A specific embodiment of the system is described below.

[0552] System configuration

[0553] The system consists of the following main components:

[0554] 1. User devices: Including smartphones, PCs, smart devices, wearable devices, etc. Data is uploaded from here.

[0555] 2. Server: The main component that analyzes materials, generates scenarios, synthesizes voice, generates visual materials, and generates videos.

[0556] 3. Network: Responsible for data communication between user terminals and servers.

[0557] Key Features

[0558] Upload materials:

[0559] Materials (PDF or text files) are uploaded to the server from the user's device. Users select and upload materials using a browser-based interface.

[0560] Content analysis of the material:

[0561] The server analyzes the uploaded documents, extracting text data using a PDF parser or text parser, and extracting important keywords and sentences using natural language processing (NLP). It also converts text in images into text using OCR technology, if necessary.

[0562] Generate an explanatory scenario:

[0563] The server generates a commentary scenario based on the analyzed text data, which determines the flow of the video and defines the insertion points for the narration content and necessary visual materials.

[0564] Narration audio and visual material creation:

[0565] The server uses Text-to-Speech (TTS) technology to create narration based on the generated scenario, and generates visual materials (illustrations, diagrams, etc.) using image generation AI and image editing tools.

[0566] Video Generation:

[0567] The server generates a video by integrating the narration audio and visual materials. Using a video editing engine (such as FFmpeg or OpenShot), the materials are placed on a timeline and transition effects and text overlays are added to create a visually appealing video.

[0568] Customize and view the generated video:

[0569] The generated videos are customized for the logistics center and displayed on smart terminals and wearable devices, allowing workers to easily review the videos and apply them to their work.

[0570] Preview and fix:

[0571] The user can preview the generated video and check it in their browser. The server provides a preview function that allows the user to play the video, check the content, and then provide feedback on any necessary corrections.

[0572] Final video credit:

[0573] Users can download the final video or generate a share link to share it with others. The server generates a secure download link, allowing users to save the video locally.

[0574] Specific examples

[0575] For example, if you upload an operating procedure manual (PDF) for a newly introduced packaging machine, the system will analyze the contents and generate a step-by-step scenario like the one below.

[0576] "Step 1: Power on the machine."

[0577] "Step 2: Select the appropriate settings in the panel."

[0578] "Step 3: Place the item in the machine."

[0579] "Step 4: Press the Start button."...and so on.

[0580] The training video is then created by synthesizing the voice and inserting images and diagrams corresponding to each step.

[0581] Prompt Sentence Examples

[0582] Below are prompts for inputting new information into a generative AI model.

[0583] Analyze the following procedure, generate narration text that clearly explains each step, and generate a scenario for creating a training video based on that.

[0584] Instructions:

[0585] Turn on the machine.

[0586] Select the appropriate settings in the panel.

[0587] Place the product in the machine.

[0588] Press the start button.

[0589] Make sure the product is packaged.

[0590] Stop the machine.

[0591] By following the above steps, it is possible to efficiently generate training videos for logistics centers.

[0592] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0593] Step 1:

[0594] This is the process of uploading materials from a user's device. The user uses a browser-based interface to select materials such as PDF or text files and clicks the upload button. The input is the file specified by the user, and the output is the file transfer to the server. This operation sends the selected materials to the server.

[0595] Step 2:

[0596] This is the process of analyzing the content of documents on the server. After the server receives the uploaded documents, it first extracts the text data using a PDF parser or text parser. The input is the uploaded document, and the output is the extracted text data. At the same time, OCR technology is used to extract the text information from the image, thereby extracting pure text information.

[0597] Step 3:

[0598] This is a process that generates an explanatory scenario based on the text data analyzed by the server. The server uses natural language processing (NLP) technology to extract keywords and important sentences from the text and analyze the structure of the material. The input is the extracted text data, and the output is an explanatory scenario. The scenario defines the flow of the video.

[0599] Step 4:

[0600] This is a process that creates narration audio based on a scenario generated on the server. The server uses Text-to-Speech (TTS) technology to convert the scenario text into audio. The input is the explanatory scenario, and the output is the narration audio file. This generates audio data corresponding to the scenario.

[0601] Step 5:

[0602] At the same time, the server creates visual materials. Image generation AI and image editing tools are used to create the necessary illustrations and diagrams based on the explanatory scenario. The input is the explanatory scenario, and the output is the visual materials. This generates image data to visually complement the data.

[0603] Step 6:

[0604] This is the process of integrating narration audio and visual materials on the server to generate a video. Using a video editing engine (such as FFmpeg or OpenShot), the materials are placed on a timeline and transition effects and text overlays are added. The input is the narration audio and visual materials, and the output is the generated video file.

[0605] Step 7:

[0606] This process customizes the generated video for the logistics center and displays it on a smart terminal or wearable device so that workers can review it. The input is the generated video file, and the output is the video displayed on a smart terminal or wearable device. This allows workers to learn their work while watching the video on-site.

[0607] Step 8:

[0608] This process previews the generated video and accepts correction instructions. The user plays the video in a browser and provides feedback on any necessary corrections. The input is the generated video file, and the output is correction instructions from the user.

[0609] Step 9:

[0610] This is the process of providing the final generated video to the user. The server generates a secure download link so that the user can save the video locally, and also generates a sharing link so that the video can be shared with other people. The input is the generated video file, and the output is the download link or sharing link.

[0611] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0612] This invention is a system that uploads materials, analyzes their contents, generates videos, and provides them in an optimal format while recognizing the user's emotions. The specific program processing flow and system operation are described in detail below.

[0613] Uploading and Loading Materials

[0614] The user uploads the selected material (PDF, text file, etc.) to the system. They open a web browser on their device, select the material on the system's upload screen, and click the upload button. This operation sends the material to the server. Furthermore, the emotion engine recognizes the user's emotional state (happiness, sadness, surprise, etc.) during uploading using a facial recognition system or voice input.

[0615] Text analysis of materials

[0616] The server receives and saves the uploaded file. It checks the file format and prepares to analyze the content of the document using a PDF parser or text parser. The content of the document is then analyzed. In the case of a PDF, the PDF parser is used to extract text data, and OCR technology is used to convert the characters in the image into text. In the case of a text file, the text data is read directly. Natural language processing (NLP) technology is then used to extract key information (keywords, important sentences, paragraphs, etc.). Based on the analyzed text data, the emotion engine adjusts the content output according to the user's emotional state (e.g., joy, sadness, surprise, etc.).

[0617] Scenario generation based on the content of the materials

[0618] The server generates a scenario for the commentary based on the analyzed text data. This scenario includes the flow of the commentary, narration content, key points, and insertion points for visual materials. Furthermore, an emotion engine evaluates the user's emotions and adjusts the content of the scenario accordingly. For example, if the user is having fun, the tone will be changed to a slightly more casual one, and if the user is sad, the explanation will be given in a warmer tone.

[0619] Voice synthesis and visual material creation

[0620] The server creates narration audio and visual materials based on the generated scenario. Text-to-speech (TTS) technology is used to convert text into audio and create the narration. At the same time, visual materials (illustrations, diagrams, etc.) are generated using image generation AI and image editing tools. The emotion engine selects the audio tone and visual style (e.g., softer colors and designs) that match the user's emotional state.

[0621] Video Generation

[0622] The server generates a video by integrating the narration audio and visual materials. It then uses a video editing engine (e.g., FFmpeg or OpenShot) to place the audio files and visual materials on a timeline. It also adds transition effects and text overlays to improve the visual quality of the video. Based on instructions from the emotion engine, it also adds effects and music that match the user's emotions.

[0623] Preview and edit your video

[0624] The server provides the generated video for the user to preview. The user then checks the video on their browser to see if the content matches their intentions. At this time, the emotion engine recognizes the user's emotions again and evaluates their reaction to the video content. If necessary, the user can specify areas to be corrected and provide feedback. The re-edited parts are adjusted taking into account the user's emotional state.

[0625] Download or share the video

[0626] The server delivers the final video that the user approves. The user can then download the video or generate a sharing link to share it with others. The server generates a secure download link so the user can save the video locally. It also generates a viewing link for internal or external sharing, and provides email and QR code generation functions as needed.

[0627] As a concrete example, consider the case where a parent creates an instructional video for their child about "constellations in space." When the parent uploads the material, the emotion engine recognizes the parent's emotions and detects that the parent is having fun. The server creates an instructional scenario with a fun, casual tone and selects bright colors and friendly visual materials. The parent previews the generated video and makes fine adjustments based on the parent's emotional response. After the corrections are reflected, the parent can download the final video and show it to their child.

[0628] This system allows users to quickly convey accurate and easy-to-understand information in a video format that best suits their emotional state. Taking emotions into consideration leads to more effective communication.

[0629] The processing flow will be explained below.

[0630] Step 1:

[0631] The user uploads the selected material (PDF, text file, etc.) to the system. They open a web browser on their device, select the material on the system's upload screen, and click the upload button. At this time, the emotion engine recognizes the user's emotional state from their facial expressions and voice.

[0632] Step 2:

[0633] The server receives and stores the uploaded file, checks the file format, and prepares the content for analysis using a PDF parser or text parser.

[0634] Step 3:

[0635] The server analyzes the content of the document. In the case of a PDF, it uses a PDF parser to extract text data and OCR technology to convert the characters in the image into text. In the case of a text file, it reads the text data directly. Next, it uses natural language processing (NLP) technology to extract key information (keywords, important sentences, paragraphs, etc.).

[0636] Step 4:

[0637] The server generates explanatory scenarios based on the analyzed text data and the user's emotional state. For example, if the user looks happy, it generates a scenario with a casual and cheerful tone, and if the user looks sad, it generates a scenario with a gentler and warmer tone.

[0638] Step 5:

[0639] The server creates narration audio and visual materials based on the scenario. It uses text-to-speech (TTS) technology to convert the scenario text into narration audio. At the same time, it follows instructions from the emotion engine to create visual materials with designs and colors that match the user's emotional state. For example, bright colors are used for cheerful emotions, and softer colors are used for calm emotions.

[0640] Step 6:

[0641] The server generates a video by integrating the narration audio and visual materials. A video editing engine (e.g., FFmpeg or OpenShot) is used to place the audio files and visual materials on a timeline. Furthermore, transition effects and text overlays are added to improve the visual quality of the video. The emotion engine also selects effects and background music that match the user's emotions.

[0642] Step 7:

[0643] The server provides the generated video for the user to preview. The user can then view the video in their browser and check whether the content matches their intent. At this time, the emotion engine again recognizes the user's emotions and evaluates their reaction to the video.

[0644] Step 8:

[0645] The user provides feedback on the parts they want to edit, and the server receives the instructions. The emotion engine re-evaluates the user's emotional state, makes any necessary adjustments, and re-edits the video. The re-edited video is then provided for preview again, and the user makes a final confirmation.

[0646] Step 9:

[0647] The server delivers the final video that the user approves. The user can then download the video or generate a sharing link to share it with others. The server generates a secure download link so the user can save the video locally. It also generates a viewing link for internal or external sharing, and provides email and QR code generation functions as needed.

[0648] This series of steps allows users to easily create and quickly deliver accurate, easy-to-understand explainer videos that best suit their emotional state, making information delivery more effective and personalized.

[0649] Example 2

[0650] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0651] In recent years, the demand for digital content has increased, and there is a particular need for the generation of video content that is optimized for individual users' emotions. However, conventional systems have had difficulty providing adaptive video content that takes into account user emotions throughout the entire process: uploading materials, analyzing content, generating scenarios, creating narration audio, creating visual materials, generating videos, and previewing and editing. This has resulted in a poor user experience and made it difficult to achieve effective communication.

[0652] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0653] In this invention, the server includes means for uploading materials, means for analyzing the content of the uploaded materials, means for generating an explanatory scenario based on the analyzed content, means for creating a narration voice and visual materials based on the generated scenario, means for generating a video by integrating the narration voice and visual materials, means for previewing the generated video and accepting correction instructions, means for providing the final generated video to a user, means for recognizing the user's emotions using a face recognition system or voice input when uploading and previewing the materials, and means for adjusting the content of the explanatory scenario and visual materials based on the user's emotions, thereby enabling the rapid and effective creation of video content optimized for the user's emotional state.

[0654] "Means for uploading materials" refers to the technical means that provides an interface for users to send materials they possess to the system and store them on the server.

[0655] "Means for analyzing the content of uploaded materials" refers to the technical means by which the server extracts and understands the content of the materials received by the server using text analysis and image processing technology.

[0656] The "means for generating an explanatory scenario" is a technical means for automatically creating a scenario including explanations, narration flow, important points, etc. based on the analyzed content.

[0657] "Means for creating narration audio and visual materials" refers to the technical means for generating narration audio and visual materials using voice synthesis technology and image generation technology based on the generated scenario.

[0658] "Means for generating a video by integrating narration audio and visual materials" refers to means for combining narration audio and visual materials into a single video file, and including video editing technology for improving editing quality.

[0659] The "means for previewing the video and accepting correction instructions" refers to a technical means for providing the generated video so that the user can check it, and accepting correction instructions from the user as necessary.

[0660] "Means for providing the final video to the user" refers to the technical means for providing the final video approved by the user in a downloadable format or as a shareable link.

[0661] A "facial recognition system" is a technology that analyzes a user's facial image and identifies a specific emotional state.

[0662] "Voice input" is a technology that analyzes a user's voice, converts the content into text, and identifies the user's emotional state from the voice.

[0663] "Means for adjusting the content of explanatory scenarios and visual materials based on the user's emotions" refers to technical means for adaptively adjusting the tone and style of the scenarios and visual materials to be generated using the user's emotional data.

[0664] This invention is a system that uploads materials, analyzes their contents, generates videos, and provides them in the most appropriate format while recognizing the user's emotions. The program processing flow and operation of this system are described in detail below.

[0665] Uploading and Loading Materials

[0666] The user uploads the selected material (PDF, text file, etc.) to the system. They open a web browser on their device, select the material on the system's upload screen, and click the "Upload" button. This operation sends the material to the server. Furthermore, the emotion engine recognizes the user's emotional state (happiness, sadness, surprise, etc.) during uploading using a facial recognition system or voice input.

[0667] Text analysis of materials

[0668] The server receives and saves the uploaded file. It checks the file format and prepares the content for analysis using a PDF parser or text parser. The content of the document is then analyzed. For PDFs, a PDF parser (e.g., PyMuPDF or PDFMiner) is used to extract text data, and OCR technology (e.g., Tesseract OCR) is used to convert the characters in the image into text. For text files, the text data is read directly. Next, natural language processing (NLP) technology (e.g., spaCy or NLTK) is used to extract key information (keywords, important sentences, clauses, etc.). Based on the analyzed text data, the emotion engine adjusts the content output according to the user's emotional state (e.g., joy, sadness, surprise, etc.).

[0669] Scenario generation based on the content of the materials

[0670] The server generates a scenario for the commentary based on the analyzed text data. This scenario includes the flow of the commentary, narration content, key points, and insertion points for visual materials. Furthermore, an emotion engine evaluates the user's emotions and adjusts the content of the scenario accordingly. For example, if the user is having fun, the tone will be changed to a slightly more casual one, and if the user is sad, the explanation will be given in a warmer tone.

[0671] Voice synthesis and visual material creation

[0672] The server creates narration audio and visual materials based on the generated scenario. It uses text-to-speech (TTS) technology (e.g., Google Text-to-Speech or Amazon Polly) to convert text into speech and create the narration. At the same time, visual materials (e.g., illustrations, diagrams, etc.) are generated using image generation AI (e.g., DALL-E or MidJourney) or image editing tools (e.g., Photoshop). The emotion engine selects the voice tone and visual style (e.g., softer colors and designs) that match the user's emotional state.

[0673] Video Generation

[0674] The server generates a video by integrating the narration audio and visual materials. It uses a video editing engine (e.g., FFmpeg or OpenShot) to place the audio files and visual materials on a timeline. It also adds transition effects and text overlays to improve the visual quality of the video. Based on instructions from the emotion engine, it also adds effects and music that match the user's emotions.

[0675] Preview and edit your video

[0676] The server provides the generated video for the user to preview. The user then checks the video on their browser to see if the content matches their intentions. At this time, the emotion engine recognizes the user's emotions again and evaluates their reaction to the video content. If necessary, the user can specify areas to be corrected and provide feedback. The re-edited parts are adjusted taking into account the user's emotional state.

[0677] Download or share the video

[0678] The server delivers the final video that the user approves. The user can then download the video or generate a sharing link to share it with others. The server generates a secure download link so the user can save the video locally. It also generates a viewing link for internal or external sharing, and provides email and QR code generation functions as needed.

[0679] Examples of concrete examples and prompts

[0680] For example, consider a case where a parent creates an instructional video for their child about "constellations in space." When the parent uploads the material, the emotion engine recognizes the parent's emotions and detects that the parent is having fun. The server creates an instructional scenario with a fun, casual tone and selects bright colors and friendly visual materials. The parent can preview the generated video and make fine adjustments based on the parent's emotional response. After the corrections are reflected, the parent can download the final video and show it to their child.

[0681] Examples of prompts include:

[0682] "How can I upload a PDF file, detect the emotion of joy, and generate a casual explainer video based on that?"

[0683] "Please explain the steps for users to upload materials online, perform real-time emotion recognition, and use that data to create a video."

[0684] This system allows users to quickly convey accurate and easy-to-understand information in a video format that best suits their emotional state. Taking emotions into consideration leads to more effective communication.

[0685] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0686] Step 1:

[0687] The user uploads the material.

[0688] Input: User selected material (PDF, text file, etc.).

[0689] Operation: Open the device's web browser, select the file on the system's upload screen, and click the "Upload" button. Emotional data is also acquired through the user's facial recognition system and voice input.

[0690] Output: The selected material file is sent to the server, along with the user's emotion data.

[0691] Step 2:

[0692] The server receives the material.

[0693] Input: Uploaded data files, user emotion data.

[0694] Operation: The server saves the received file in the specified directory. It determines the file format (PDF, text file, etc.).

[0695] Output: The path of the saved document file, file format information.

[0696] Step 3:

[0697] The server analyzes the data.

[0698] Input: Saved material file, file format information.

[0699] How it works: For PDF files, the server extracts text using a PDF parser (e.g., PyMuPDF or PDFMiner) and converts characters in images to text using OCR technology (e.g., Tesseract OCR). For text files, the server reads the text data directly and extracts key information using natural language processing (NLP) technology (e.g., spaCy or NLTK).

[0700] Output: Parsed text data, extracted key information.

[0701] Step 4:

[0702] The emotion engine evaluates the user's emotions.

[0703] Input: User emotion data, parsed text data, extracted key information.

[0704] How it works: The emotion engine evaluates the analyzed text data based on the user's emotional state (happiness, sadness, surprise, etc.) and adjusts the content output.

[0705] Output: Emotion evaluation results, adjusted content output instructions.

[0706] Step 5:

[0707] The server generates the scenario.

[0708] Input: Parsed text data, extracted key information, sentiment evaluation results, and tailored content output instructions.

[0709] Operation: Based on these input data, the server generates a commentary scenario, which includes the flow of the commentary, narration content, important points, and insertion points for visual materials.

[0710] Output: The generated explanatory scenario.

[0711] Step 6:

[0712] The server creates the audio and visual material.

[0713] Input: Generated commentary scenario, emotion evaluation results.

[0714] How it works: The server uses voice synthesis technology (e.g., Google Text-to-Speech or Amazon Polly) to create the narration, and image generation AI (e.g., DALL-E or MidJourney) or image editing tools (e.g., Photoshop) to generate the visuals. The voice tone and visual style are adjusted to match the user's emotional state.

[0715] Output: Generated narration audio and visual materials.

[0716] Step 7:

[0717] The server generates the video.

[0718] Input: Generated narration audio, visual materials, and emotion evaluation results.

[0719] How it works: The server uses a video editing engine (e.g., FFmpeg or OpenShot) to combine audio and visual material into a single video file. It also adds transition effects and text overlays to enhance the visual quality. It also adds effects and music that suit the user's emotions.

[0720] Output: The generated video file.

[0721] Step 8:

[0722] The server provides a preview of the generated video.

[0723] Input: Generated video files, emotion evaluation results.

[0724] How it works: The server provides the user with a link to preview the video in their browser. The user reviews the video content and makes corrections as needed. The emotion engine again recognizes the user's emotions and evaluates their reactions during the preview.

[0725] Output: User's correction instructions, updated emotion evaluation results.

[0726] Step 9:

[0727] The server provides the final video.

[0728] Input: Generated video file, correction instructions from user.

[0729] How it works: The server re-edits the video based on the correction instructions and generates the final version. It then provides the final video to the user and generates a download link and a sharing link.

[0730] Output: Final video file, secure download link, and share link.

[0731] (Application example 2)

[0732] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0733] In conventional video generation systems, the process of analyzing the content of materials and creating videos is uniformly created without considering the user's emotional state, making it difficult to provide personalized content. Furthermore, in order for users to optimally understand and enjoy the content of videos, adjustments based on the user's emotions are necessary, but there has been a lack of systems that can achieve this.

[0734] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0735] In this invention, the server includes means for uploading materials, means for analyzing the content of the uploaded materials, means for generating an explanatory scenario based on the analyzed content, means for creating a narration voice and visual materials based on the generated scenario, means for generating a video by integrating the narration voice and visual materials, means for previewing the generated video and accepting correction instructions, means for recognizing the emotional state of the user and adjusting the scenario and visual style according to the emotion, and means for providing the user with the final generated video, thereby making it possible to provide optimal content according to the user's emotion.

[0736] "Means for uploading materials" refers to a device or method for uploading materials owned by a user into the system.

[0737] "Means for analyzing the content of uploaded material" means a device or method used to computerize and understand the content of uploaded material.

[0738] The "means for generating a commentary scenario" is a device or method for creating a plot or text for explanation or narration based on the analyzed content.

[0739] The "means for creating narration audio and visual materials" refers to a device or method for creating audio data and visual materials based on a generated scenario.

[0740] "Means for generating video" refers to a device or method for combining narration audio and visual materials to edit and generate a single video.

[0741] The "means for previewing the generated video and accepting correction instructions" refers to a device or method that allows a user to view the generated video in advance and indicate any necessary corrections.

[0742] "Means for recognizing the user's emotional state and adjusting the scenario and visual style according to that emotion" refers to a device or method that reads the user's emotions from their facial expressions, voice, etc., and changes or adjusts the content of the scenario and the visual design based on that.

[0743] The "means for providing the final video to the user" refers to a device or method that allows the user to download the completed video or share it with others.

[0744] An embodiment of the present invention will be described. This system analyzes materials uploaded by users and generates optimal explanatory videos while recognizing their emotional state. This system makes it possible to provide personalized video content that reflects the emotions of each individual user.

[0745] Uploading and Loading Materials

[0746] Users access the system's upload screen using a smartphone or computer browser. On the upload screen, users select and upload documents in PDF or text file format. At the same time as uploading, the system uses the device's built-in camera and microphone to recognize the user's emotional state through a facial recognition system or voice input. This emotion recognition uses facial expression recognition software and voice analysis software.

[0747] Text analysis of materials

[0748] The server receives the uploaded documents and extracts the text data using a PDF parser and OCR technology. This converts the text information from the images in the documents into text. Key information is then extracted using natural language processing (NLP). At this stage, the server also adjusts the analysis results based on the user's emotional state.

[0749] Scenario Generation

[0750] The server generates a commentary scenario for the user based on the emotional state and the analyzed text data. The generated scenario includes the flow of the commentary, the narration content, and the insertion points of visual materials. The emotion engine evaluates the user's emotions and adjusts the content of the scenario. For example, if the user is having fun, it will use casual language and select visually bright materials.

[0751] Creation of narration audio and visual materials

[0752] The server uses text-to-speech (TTS) technology to generate narration based on the generated scenario. At the same time, it creates visual materials such as illustrations and diagrams using generative AI models and image editing tools. At this stage, it also selects the style and tone that best suits the user's emotions.

[0753] Video Generation

[0754] The server combines the generated narration audio with the visual material and generates a video using a video editing engine (e.g., FFmpeg or OpenShot). The narration and visual material are placed on a timeline, and the visual quality is improved by adding transition effects and text overlays. Again, visual effects and music are applied based on the instructions of the emotion engine.

[0755] Preview and edit your video

[0756] The video generated by the server is previewed by the user. The user checks the video in their browser to ensure that the content is as intended. At the same time, the emotion engine evaluates the user's emotional response to the video. If necessary, the user can specify corrections and the video is re-edited.

[0757] Download or share the video

[0758] The final video is then delivered to the user via the server, who can either download the video or generate a shareable link to share it with others. The server generates a secure download link and also provides a viewing link for internal or external sharing.

[0759] Specific examples

[0760] For example, if a parent wants to create a learning video for their child on "Earth Geography," the system will automatically detect the parent's emotional state when they upload the materials. If a happy emotion is detected, a casual and fun-toned commentary scenario will be generated. Brightly colored diagrams and animations will be selected, and the parent can fine-tune the video while previewing it. Finally, a fun-toned geography learning video will be generated, which the parent can show to their child.

[0761] Prompt Sentence Examples

[0762] "Analyze this material and generate fun narration and colorful visuals to create a geography learning video for children."

[0763] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0764] Step 1:

[0765] Users access the system's upload screen using their devices, select and upload documents in PDF or text file format. The input data is the document file owned by the user, and the output data is the document file sent to the server. The user's emotional state is also collected at the same time using a facial recognition system and voice input. The input data is the user's facial expressions and voice, and the output data is the emotion recognition results.

[0766] Step 2:

[0767] The server analyzes the received document file. In the case of PDF, it uses a PDF parser to extract the text data and OCR technology to convert the characters in the image into text. The input data is the document file sent to the server, and the output data is the extracted text data.

[0768] Step 3:

[0769] The server uses natural language processing (NLP) techniques to extract keywords and important sentences from the extracted text data. The input data is the extracted text data, and the output data is a list of keywords and important sentences.

[0770] Step 4:

[0771] The server generates an explanatory scenario based on the emotion recognition results and the analyzed text data. The input data is a list of keywords and important sentences and the emotion recognition results, and the output data is the generated explanatory scenario. The emotion recognition results affect the tone and content of the scenario. For example, a casual scenario is generated for a positive emotional state.

[0772] Step 5:

[0773] The server uses text-to-speech (TTS) technology to create a narration voice based on the generated scenario. The input data is the generated commentary scenario, and the output data is a narration voice file. The TTS engine converts the text into speech.

[0774] Step 6:

[0775] The server creates visual materials using generative AI models and image editing tools. The input data is the generated explanatory scenario, and the output data is a set of visual materials. Diagrams and illustrations are generated to match the key points of the scenario.

[0776] Step 7:

[0777] The server uses a video editing engine (e.g., FFmpeg or OpenShot) to combine the narration audio and visual materials to generate a video. The input data is a narration audio file and a set of visual materials, and the output data is the generated video file. Transition effects and text overlays are also added.

[0778] Step 8:

[0779] The server allows the user to preview the generated video. The user checks the video on the browser and provides feedback on any corrections. The input data is the generated video file, and the output data is the user's feedback. The emotion engine reevaluates the user's emotions about the video and identifies areas that need to be re-edited.

[0780] Step 9:

[0781] The server re-edits the video based on the user's feedback and makes final adjustments according to the emotions. The input data are the user's feedback and the emotion re-evaluation results, and the output data is the final video file.

[0782] Step 10:

[0783] The server provides the final video to the user, who can then download it or share it with others via a sharing link. The input data is the final video file, and the output data is a download link and a sharing link. The server generates a secure download link and provides a viewing link.

[0784] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0785] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0786] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0787] [Third embodiment]

[0788] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0789] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0790] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0791] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0792] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0793] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0794] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0795] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0796] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0797] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0798] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0799] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0800] This invention is a system that uploads materials, analyzes the contents, generates videos, and provides them to users. The specific program processing flow and system operation are described in detail below.

[0801] Uploading and Loading Materials

[0802] The user uploads the selected book, dictionary, picture book, or material (PDF, text file, etc.) to the system. Using the device's web browser, the user selects the material on the designated upload screen and clicks the upload button. This operation sends the material to the server.

[0803] Text analysis of materials

[0804] The server analyzes the uploaded document. It uses a PDF parser and text parser to extract the text data from the document. It then uses natural language processing (NLP) technology to extract keywords and important sentences from the text and analyze the structure of the document. If necessary, OCR (optical character recognition) technology is used to convert text in images into text.

[0805] Scenario generation based on the content of the materials

[0806] The server generates a commentary scenario based on the analyzed text data. This is the process of determining the flow of the video and defining the narration content and insertion points for necessary visual materials. Structured data is created, specifying the commentary order and detailed narration content for each section.

[0807] Voice synthesis and visual material creation

[0808] The server creates narration audio and visual materials based on the generated scenario. It uses Text-to-Speech (TTS) technology to convert text into audio and create the narration. At the same time, visual materials (illustrations, diagrams, etc.) are generated using image generation AI and image editing tools.

[0809] Video Generation

[0810] The server combines the narration audio with the visual material to generate the video. Using a video editing engine (such as FFmpeg or OpenShot), the material is placed on a timeline and transition effects and text overlays are added to create a visually appealing video. At this stage, the video is exported to its final form and saved in an appropriate format, such as MP4.

[0811] Preview and edit your video

[0812] The user can preview the generated video and check it on the browser. The server provides a preview function that enables this, and after the user plays the video and checks the content, they can provide feedback on any necessary corrections. Upon receiving the correction request, the server re-edits the relevant parts and updates the video.

[0813] Download or share the video

[0814] Users can download the final video or generate a sharing link to share it with others. The server generates a secure download link, allows users to save the video locally, and generates a viewing link for internal or external sharing, and also provides the ability to send an email or generate a QR code if desired.

[0815] As a concrete example, let's consider the case where a parent creates an explanatory video for their child about the planets of the solar system. The parent uploads a PDF of an elementary school science textbook and inputs it into the system. The server analyzes the material and extracts information about the planets in the solar system. An explanatory scenario is then generated, and narration audio and planet illustrations are created. These materials are then integrated into a video. The parent can use the preview function to check the video and make any final corrections. After the corrections are complete, the parent can download the final video and show it to their child.

[0816] Through this series of steps, users can easily provide accurate information in video format, saving a great deal of time and effort. By using this system, information can be transmitted quickly and effectively.

[0817] The processing flow will be explained below.

[0818] Step 1:

[0819] The user uploads the selected material (PDF, text file, etc.) to the system by opening a web browser on the terminal, selecting the material on the system's upload screen, and clicking the upload button.

[0820] Step 2:

[0821] The server receives and stores the uploaded file, first checking the file format and preparing to parse the content using a PDF parser or text parser.

[0822] Step 3:

[0823] The server analyzes the content of the document. In the case of a PDF, it uses a PDF parser to extract the text data and OCR technology to convert the characters in the image into text. In the case of a text file, it reads the text data directly. Next, it uses natural language processing (NLP) technology to extract key information (keywords, important sentences, paragraphs, etc.).

[0824] Step 4:

[0825] The server generates a commentary scenario based on the analyzed text data. The scenario includes the flow of the commentary, narration content, important points, and points where visual materials should be inserted. The scenario is organized as structured data.

[0826] Step 5:

[0827] The server creates narration audio and visual materials based on the scenario. It uses text-to-speech (TTS) technology to convert the scenario text into narration audio. At the same time, it uses image generation AI and image editing tools to generate the necessary visual materials, such as illustrations and diagrams.

[0828] Step 6:

[0829] The server generates a video by integrating the narration audio and visual materials. Using a video editing engine (e.g., FFmpeg or OpenShot), the audio files and visual materials are placed on a timeline. Furthermore, transition effects and text overlays are added to improve the visual quality of the video.

[0830] Step 7:

[0831] The server provides the generated video so that the user can preview it. The user can check the video in the browser and check whether the content matches the user's intention. If necessary, the user can specify corrections and provide feedback.

[0832] Step 8:

[0833] The server receives correction instructions from the user, re-edits the relevant parts, and provides the re-edited video for preview again so that the user can make a final check.

[0834] Step 9:

[0835] The server then provides the final video that the user has approved. The user can then download the video or generate a sharing link to share it with others. The server generates secure download links and viewing links, and also provides email and QR code generation functions as needed.

[0836] By following the above steps, users can easily create and quickly provide accurate and easy-to-understand instructional videos based on materials of their own choosing. This system can be used for a wide range of purposes, such as home education and creating business manuals in companies.

[0837] Example 1

[0838] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0839] Conventional document analysis systems often lack sufficient methods for efficiently understanding the content of documents. In particular, it is difficult to quickly and accurately analyze vast amounts of data and provide the information in a format that is easy for users to understand, which requires time and effort. Furthermore, simply providing the content of documents in text format makes it difficult to convert it into a visually understandable format, making it ineffective for use in education, presentations, and other applications. Therefore, there is a growing demand for a system that can analyze uploaded documents and provide information in video format based on the results.

[0840] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0841] In this invention, the server includes means for uploading materials, means for analyzing the content of the uploaded materials, means for generating an explanatory scenario based on the analyzed content, means for creating a narration voice and visual materials based on the generated scenario, means for generating a video by integrating the narration voice and visual materials, means for previewing the generated video and accepting correction instructions, and means for providing the final generated video to the user. This makes it possible to analyze the content of the uploaded materials and provide the information quickly and efficiently in a video format that is easy for the user to understand.

[0842] The "means for uploading materials" is a function for transmitting materials selected by the user to the system and storing them on the server.

[0843] "Means for analyzing the content of uploaded materials" refers to a function for extracting text data and other information from uploaded materials and converting that information into an easily understandable format.

[0844] The "means for generating an explanatory scenario" is a function that creates a scenario for determining the structure of the explanatory video and the content of the narration based on the content of the analyzed materials.

[0845] "Means for creating narration audio and visual materials" refers to a function for creating narration audio and visual materials using voice synthesis technology and image generation technology in accordance with the generated scenario.

[0846] "Means for integrating narration audio and visual materials to generate video" refers to a function that combines the created audio and visual materials on a timeline and outputs them as a completed video.

[0847] The "means for previewing the generated video and accepting correction instructions" is a function that allows the user to check the generated video and provide feedback on parts that need correction.

[0848] The "means for providing the final video to the user" is a function that allows the user to download the finalized video or generate a link that allows the user to share it with other people.

[0849] "Natural language processing technology" is a method for analyzing text data, extracting keywords and important sentences from it, and understanding the content of the material.

[0850] "Speech synthesis technology" is a technology that generates synthetic speech based on text data.

[0851] "Image generation technology" refers to the technology of creating visual materials based on text or other data.

[0852] A "video editing engine" is software or a system for generating and editing videos by combining narration audio and visual materials.

[0853] The present invention provides a system for uploading materials, analyzing the contents of the materials, generating videos, and providing the videos to users. Specific embodiments will be described in detail below.

[0854] Uploading and Loading Materials

[0855] The user accesses the system's upload screen using a web browser on their device and uploads the selected materials (PDF, text files, etc.). The user clicks the upload button, which sends the materials to the server.

[0856] Analysis of data

[0857] The server receives the uploaded documents and temporarily stores them in storage. It then uses a PDF parser (pdfminer.six) to extract text data from the PDF file. For text files, it directly reads the content. It then uses natural language processing (NLP) technology (NLTK) to extract keywords and important sentences from the text and analyze the structure of the document. If necessary, it also converts text within images into text using OCR (optical character recognition) technology (Tesseract OCR).

[0858] Generate a scenario

[0859] The server generates an explanatory scenario based on the analyzed text data. This scenario includes the flow of the video, the narration content, and insertion points for visual materials. To generate the scenario, a generative AI model is used and a prompt is entered. For example, the prompt might read, "Please create a scenario that explains the planets in the solar system for elementary school science classes. I have uploaded a PDF of a science textbook as a reference. Based on this reference, please generate a scenario that includes the characteristics, relative positions, and important information about each planet."

[0860] Voice synthesis and visual material creation

[0861] Based on the generated scenario, the server uses text-to-speech (TTS) technology (Google Text-to-Speech API) to convert text into speech and create the narration. At the same time, it uses image generation AI (e.g., DALL-E) and image editing tools (Adobe Photoshop) to create the necessary visual materials.

[0862] Video generation

[0863] The server integrates the created narration audio with the visual material, and then uses a video editing engine (e.g., FFmpeg) to place the material on a timeline, adding transition effects and text overlays to create a visually appealing video.

[0864] Preview and edit your video

[0865] The user can preview the generated video on the browser and check the content. The server provides a preview function and gives feedback to the user on any corrections that need to be made. The server receives the correction request, re-edits the relevant parts, and updates the video.

[0866] Video provision

[0867] Users can then download the final video or share it with others. The server generates a secure download link and provides it to the user. It can also generate a sharing link for viewing, which can be sent via email or generated as a QR code.

[0868] As a specific example, if a parent wants to create an explanatory video for their child about the planets in the solar system, they upload a PDF of a science textbook to the system. The server analyzes the material and extracts information about the planets in the solar system. The generative AI model then generates a scenario and uses speech synthesis and image generation technologies to create narration audio and illustrations of the planets. These are then integrated to generate a video, and the user (parent) can use the preview function to check the content, make any necessary corrections, and then download or share the final video.

[0869] This system allows users to easily create high-quality videos for educational and presentation purposes.

[0870] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0871] Step 1:

[0872] The user accesses the system's upload screen using the terminal's web browser, selects the document file (PDF, text file, etc.), and clicks the upload button.

[0873] Input: Document file

[0874] Output: The document file is sent to the server.

[0875] Operation: When the user operates the application, the document file is sent to the server via an HTTP request and temporarily stored in storage.

[0876] Step 2:

[0877] The server analyzes the received data file.

[0878] Input: Uploaded file

[0879] Output: Parsed text data

[0880] Operation: For PDF files, it uses the PDF parser (pdfminer.six) to extract text data. For text files, it reads the content directly. If necessary, it also uses OCR (Tesseract OCR) technology to convert characters in images into text.

[0881] Step 3:

[0882] The server uses natural language processing (NLP) technology (NLTK) to extract keywords and important sentences from the analyzed text data.

[0883] Input: Parsed text data

[0884] Output: Extracted keywords and important sentences

[0885] How it works: NLP technology analyzes text and converts the structure of the material into an easy-to-understand format.

[0886] Step 4:

[0887] Based on the extracted information, the server uses a generative AI model to generate an explanatory scenario.

[0888] Input: Extracted keywords and important sentences

[0889] Output: Explaining scenario

[0890] How it works: A prompt for scenario generation is input into the generative AI model, which then creates a scenario that includes the flow of explanations, narration content, and insertion points for visual materials. For example, the prompt might be, "Please create a scenario that explains the planets in the solar system for use in elementary school science classes."

[0891] Step 5:

[0892] The server creates narration audio based on the generated scenario using a speech synthesis API (Google Text-to-Speech API).

[0893] Input: Explaining scenario

[0894] Output: Narration audio file

[0895] Operation: The scenario text is input into the speech synthesis API, synthetic speech is generated, and saved as a narration audio file.

[0896] Step 6:

[0897] Based on the generated scenario, the server creates visual materials using image generation AI and image editing tools.

[0898] Input: Explaining scenario

[0899] Output: Visual material (image file)

[0900] How it works: Using an image generation AI (e.g., DALL-E) or an image editing tool (Adobe Photoshop), the necessary illustrations and diagrams are generated and saved as image files.

[0901] Step 7:

[0902] The server combines the narration audio and visual material to generate a video.

[0903] Input: Narration audio file, visual material

[0904] Output: Generated video file

[0905] How it works: Use a video editing engine (e.g. FFmpeg) to combine audio and visual elements on the timeline, add transition effects and text overlays to create a video, and save the finished video in an appropriate format, such as MP4.

[0906] Step 8:

[0907] The user can preview the generated animation and provide any necessary corrections.

[0908] Input: Generated video file

[0909] Output: Correction instructions

[0910] How it works: The user plays the video and provides feedback using the preview functionality provided by the server. The server receives correction requests, re-edits the relevant parts, and updates the video.

[0911] Step 9:

[0912] Users can then download the final edited video or generate a shareable link to share it with others.

[0913] Input: Final generated video

[0914] Output: Download link or share link

[0915] How it works: The server generates a secure download link and provides it to the user. The user can then download the video or share it with others using a sharing link. The server also provides the ability to email the sharing link or generate a QR code.

[0916] (Application example 1)

[0917] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0918] Logistics centers need a method to efficiently and quickly communicate work procedures and equipment operation methods to workers. Traditionally, manuals using text and still images have been widely used, but they have problems such as being difficult to understand and being difficult to update. In addition, when new work procedures or equipment are introduced, training is required, which takes time and effort. To solve these problems, a more effective and easy-to-understand method is needed.

[0919] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0920] In this invention, the server includes means for uploading materials, means for analyzing the contents of the uploaded materials, means for generating an explanatory scenario based on the analyzed contents, means for creating a narration voice and visual materials based on the generated scenario, means for generating a video by integrating the narration voice and visual materials, means for customizing the generated video for the logistics center and displaying it on a smart terminal or a wearable device so that workers can check it, means for previewing the generated video and accepting correction instructions, and means for providing the final generated video to the user. This makes it possible to provide work procedures and equipment operation methods in a video format that is instantly easy to understand.

[0921] "Documents" are document data uploaded by users, such as PDFs or text files.

[0922] "Uploading means" refers to a means that has the function of allowing a user to transfer materials to a server via a network.

[0923] "Means for analyzing" means means including technology for analyzing the content of uploaded materials and converting them into an understandable form.

[0924] An "explanatory scenario" is a plan that defines the flow and sequence of video and audio, generated based on the content of the analyzed materials.

[0925] "Narration voice" is voice data created based on a commentary scenario, and conveys the commentary content through sound.

[0926] "Visual materials" are visual materials such as illustrations and diagrams used to complement the explanatory scenario.

[0927] "Means for generating moving images" means means including technology for integrating narration audio and visual materials into a single moving image format.

[0928] "Customizing for the distribution center" means adjusting the format and content of the generated video to be suitable for use in the distribution center.

[0929] "Smart devices" refer to portable information terminals such as smartphones and tablets.

[0930] A "wearable device" is a device that can be worn by a worker and that displays information.

[0931] "Preview" means temporarily viewing the generated video for final confirmation.

[0932] A "correction instruction" is a request for correction or modification made by a user to a generated video.

[0933] The present invention is a system for automatically generating training videos for a logistics center, which operates mainly on a server and a user terminal. A specific embodiment of the system is described below.

[0934] System configuration

[0935] The system consists of the following main components:

[0936] 1. User devices: Including smartphones, PCs, smart devices, wearable devices, etc. Data is uploaded from here.

[0937] 2. Server: The main component that analyzes materials, generates scenarios, synthesizes voice, generates visual materials, and generates videos.

[0938] 3. Network: Responsible for data communication between user terminals and servers.

[0939] Key Features

[0940] Upload materials:

[0941] Materials (PDF or text files) are uploaded to the server from the user's device. Users select and upload materials using a browser-based interface.

[0942] Content analysis of the material:

[0943] The server analyzes the uploaded documents, extracting text data using a PDF parser or text parser, and extracting important keywords and sentences using natural language processing (NLP). It also converts text in images into text using OCR technology, if necessary.

[0944] Generate an explanatory scenario:

[0945] The server generates a commentary scenario based on the analyzed text data, which determines the flow of the video and defines the insertion points for the narration content and necessary visual materials.

[0946] Narration audio and visual material creation:

[0947] The server uses Text-to-Speech (TTS) technology to create narration based on the generated scenario, and generates visual materials (illustrations, diagrams, etc.) using image generation AI and image editing tools.

[0948] Video Generation:

[0949] The server generates a video by integrating the narration audio and visual materials. Using a video editing engine (such as FFmpeg or OpenShot), the materials are placed on a timeline and transition effects and text overlays are added to create a visually appealing video.

[0950] Customize and view the generated video:

[0951] The generated videos are customized for the logistics center and displayed on smart terminals and wearable devices, allowing workers to easily review the videos and apply them to their work.

[0952] Preview and fix:

[0953] The user can preview the generated video and check it in their browser. The server provides a preview function that allows the user to play the video, check the content, and then provide feedback on any necessary corrections.

[0954] Final video credit:

[0955] Users can download the final video or generate a share link to share it with others. The server generates a secure download link, allowing users to save the video locally.

[0956] Specific examples

[0957] For example, if you upload an operating procedure manual (PDF) for a newly introduced packaging machine, the system will analyze the contents and generate a step-by-step scenario like the one below.

[0958] "Step 1: Power on the machine."

[0959] "Step 2: Select the appropriate settings in the panel."

[0960] "Step 3: Place the item in the machine."

[0961] "Step 4: Press the Start button."...and so on.

[0962] The training video is then created by synthesizing the voice and inserting images and diagrams corresponding to each step.

[0963] Prompt Sentence Examples

[0964] Below are prompts for inputting new information into a generative AI model.

[0965] Analyze the following procedure, generate narration text that clearly explains each step, and generate a scenario for creating a training video based on that.

[0966] Instructions:

[0967] Turn on the machine.

[0968] Select the appropriate settings in the panel.

[0969] Place the product in the machine.

[0970] Press the start button.

[0971] Make sure the product is packaged.

[0972] Stop the machine.

[0973] By following the above steps, it is possible to efficiently generate training videos for logistics centers.

[0974] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0975] Step 1:

[0976] This is the process of uploading materials from a user's device. The user uses a browser-based interface to select materials such as PDF or text files and clicks the upload button. The input is the file specified by the user, and the output is the file transfer to the server. This operation sends the selected materials to the server.

[0977] Step 2:

[0978] This is the process of analyzing the content of documents on the server. After the server receives the uploaded documents, it first extracts the text data using a PDF parser or text parser. The input is the uploaded document, and the output is the extracted text data. At the same time, OCR technology is used to extract the text information from the image, thereby extracting pure text information.

[0979] Step 3:

[0980] This is a process that generates an explanatory scenario based on the text data analyzed by the server. The server uses natural language processing (NLP) technology to extract keywords and important sentences from the text and analyze the structure of the material. The input is the extracted text data, and the output is an explanatory scenario. The scenario defines the flow of the video.

[0981] Step 4:

[0982] This is a process that creates narration audio based on a scenario generated on the server. The server uses Text-to-Speech (TTS) technology to convert the scenario text into audio. The input is the explanatory scenario, and the output is the narration audio file. This generates audio data corresponding to the scenario.

[0983] Step 5:

[0984] At the same time, the server creates visual materials. Image generation AI and image editing tools are used to create the necessary illustrations and diagrams based on the explanatory scenario. The input is the explanatory scenario, and the output is the visual materials. This generates image data to visually complement the data.

[0985] Step 6:

[0986] This is the process of integrating narration audio and visual materials on the server to generate a video. Using a video editing engine (such as FFmpeg or OpenShot), the materials are placed on a timeline and transition effects and text overlays are added. The input is the narration audio and visual materials, and the output is the generated video file.

[0987] Step 7:

[0988] This process customizes the generated video for the logistics center and displays it on a smart terminal or wearable device so that workers can review it. The input is the generated video file, and the output is the video displayed on a smart terminal or wearable device. This allows workers to learn their work while watching the video on-site.

[0989] Step 8:

[0990] This process previews the generated video and accepts correction instructions. The user plays the video in a browser and provides feedback on any necessary corrections. The input is the generated video file, and the output is correction instructions from the user.

[0991] Step 9:

[0992] This is the process of providing the final generated video to the user. The server generates a secure download link so that the user can save the video locally, and also generates a sharing link so that the video can be shared with other people. The input is the generated video file, and the output is the download link or sharing link.

[0993] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0994] This invention is a system that uploads materials, analyzes their contents, generates videos, and provides them in an optimal format while recognizing the user's emotions. The specific program processing flow and system operation are described in detail below.

[0995] Uploading and Loading Materials

[0996] The user uploads the selected material (PDF, text file, etc.) to the system. They open a web browser on their device, select the material on the system's upload screen, and click the upload button. This operation sends the material to the server. Furthermore, the emotion engine recognizes the user's emotional state (happiness, sadness, surprise, etc.) during uploading using a facial recognition system or voice input.

[0997] Text analysis of materials

[0998] The server receives and saves the uploaded file. It checks the file format and prepares to analyze the content of the document using a PDF parser or text parser. The content of the document is then analyzed. In the case of a PDF, the PDF parser is used to extract text data, and OCR technology is used to convert the characters in the image into text. In the case of a text file, the text data is read directly. Natural language processing (NLP) technology is then used to extract key information (keywords, important sentences, paragraphs, etc.). Based on the analyzed text data, the emotion engine adjusts the content output according to the user's emotional state (e.g., joy, sadness, surprise, etc.).

[0999] Scenario generation based on the content of the materials

[1000] The server generates a scenario for the commentary based on the analyzed text data. This scenario includes the flow of the commentary, narration content, key points, and insertion points for visual materials. Furthermore, an emotion engine evaluates the user's emotions and adjusts the content of the scenario accordingly. For example, if the user is having fun, the tone will be changed to a slightly more casual one, and if the user is sad, the explanation will be given in a warmer tone.

[1001] Voice synthesis and visual material creation

[1002] The server creates narration audio and visual materials based on the generated scenario. Text-to-speech (TTS) technology is used to convert text into audio and create the narration. At the same time, visual materials (illustrations, diagrams, etc.) are generated using image generation AI and image editing tools. The emotion engine selects the audio tone and visual style (e.g., softer colors and designs) that match the user's emotional state.

[1003] Video Generation

[1004] The server generates a video by integrating the narration audio and visual materials. It then uses a video editing engine (e.g., FFmpeg or OpenShot) to place the audio files and visual materials on a timeline. It also adds transition effects and text overlays to improve the visual quality of the video. Based on instructions from the emotion engine, it also adds effects and music that match the user's emotions.

[1005] Preview and edit your video

[1006] The server provides the generated video for the user to preview. The user then checks the video on their browser to see if the content matches their intentions. At this time, the emotion engine recognizes the user's emotions again and evaluates their reaction to the video content. If necessary, the user can specify areas to be corrected and provide feedback. The re-edited parts are adjusted taking into account the user's emotional state.

[1007] Download or share the video

[1008] The server delivers the final video that the user approves. The user can then download the video or generate a sharing link to share it with others. The server generates a secure download link so the user can save the video locally. It also generates a viewing link for internal or external sharing, and provides email and QR code generation functions as needed.

[1009] As a concrete example, consider the case where a parent creates an instructional video for their child about "constellations in space." When the parent uploads the material, the emotion engine recognizes the parent's emotions and detects that the parent is having fun. The server creates an instructional scenario with a fun, casual tone and selects bright colors and friendly visual materials. The parent previews the generated video and makes fine adjustments based on the parent's emotional response. After the corrections are reflected, the parent can download the final video and show it to their child.

[1010] This system allows users to quickly convey accurate and easy-to-understand information in a video format that best suits their emotional state. Taking emotions into consideration leads to more effective communication.

[1011] The processing flow will be explained below.

[1012] Step 1:

[1013] The user uploads the selected material (PDF, text file, etc.) to the system. They open a web browser on their device, select the material on the system's upload screen, and click the upload button. At this time, the emotion engine recognizes the user's emotional state from their facial expressions and voice.

[1014] Step 2:

[1015] The server receives and stores the uploaded file, checks the file format, and prepares the content for analysis using a PDF parser or text parser.

[1016] Step 3:

[1017] The server analyzes the content of the document. In the case of a PDF, it uses a PDF parser to extract text data and OCR technology to convert the characters in the image into text. In the case of a text file, it reads the text data directly. Next, it uses natural language processing (NLP) technology to extract key information (keywords, important sentences, paragraphs, etc.).

[1018] Step 4:

[1019] The server generates explanatory scenarios based on the analyzed text data and the user's emotional state. For example, if the user looks happy, it generates a scenario with a casual and cheerful tone, and if the user looks sad, it generates a scenario with a gentler and warmer tone.

[1020] Step 5:

[1021] The server creates narration audio and visual materials based on the scenario. It uses text-to-speech (TTS) technology to convert the scenario text into narration audio. At the same time, it follows instructions from the emotion engine to create visual materials with designs and colors that match the user's emotional state. For example, bright colors are used for cheerful emotions, and softer colors are used for calm emotions.

[1022] Step 6:

[1023] The server generates a video by integrating the narration audio and visual materials. A video editing engine (e.g., FFmpeg or OpenShot) is used to place the audio files and visual materials on a timeline. Furthermore, transition effects and text overlays are added to improve the visual quality of the video. The emotion engine also selects effects and background music that match the user's emotions.

[1024] Step 7:

[1025] The server provides the generated video for the user to preview. The user can then view the video in their browser and check whether the content matches their intent. At this time, the emotion engine again recognizes the user's emotions and evaluates their reaction to the video.

[1026] Step 8:

[1027] The user provides feedback on the parts they want to edit, and the server receives the instructions. The emotion engine re-evaluates the user's emotional state, makes any necessary adjustments, and re-edits the video. The re-edited video is then provided for preview again, and the user makes a final confirmation.

[1028] Step 9:

[1029] The server delivers the final video that the user approves. The user can then download the video or generate a sharing link to share it with others. The server generates a secure download link so the user can save the video locally. It also generates a viewing link for internal or external sharing, and provides email and QR code generation functions as needed.

[1030] This series of steps allows users to easily create and quickly deliver accurate, easy-to-understand explainer videos that best suit their emotional state, making information delivery more effective and personalized.

[1031] Example 2

[1032] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1033] In recent years, the demand for digital content has increased, and there is a particular need for the generation of video content that is optimized for individual users' emotions. However, conventional systems have had difficulty providing adaptive video content that takes into account user emotions throughout the entire process: uploading materials, analyzing content, generating scenarios, creating narration audio, creating visual materials, generating videos, and previewing and editing. This has resulted in a poor user experience and made it difficult to achieve effective communication.

[1034] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1035] In this invention, the server includes means for uploading materials, means for analyzing the content of the uploaded materials, means for generating an explanatory scenario based on the analyzed content, means for creating a narration voice and visual materials based on the generated scenario, means for generating a video by integrating the narration voice and visual materials, means for previewing the generated video and accepting correction instructions, means for providing the final generated video to a user, means for recognizing the user's emotions using a face recognition system or voice input when uploading and previewing the materials, and means for adjusting the content of the explanatory scenario and visual materials based on the user's emotions, thereby enabling the rapid and effective creation of video content optimized for the user's emotional state.

[1036] "Means for uploading materials" refers to the technical means that provides an interface for users to send materials they possess to the system and store them on the server.

[1037] "Means for analyzing the content of uploaded materials" refers to the technical means by which the server extracts and understands the content of the materials received by the server using text analysis and image processing technology.

[1038] The "means for generating an explanatory scenario" is a technical means for automatically creating a scenario including explanations, narration flow, important points, etc. based on the analyzed content.

[1039] "Means for creating narration audio and visual materials" refers to the technical means for generating narration audio and visual materials using voice synthesis technology and image generation technology based on the generated scenario.

[1040] "Means for generating a video by integrating narration audio and visual materials" refers to means for combining narration audio and visual materials into a single video file, and including video editing technology for improving editing quality.

[1041] The "means for previewing the video and accepting correction instructions" refers to a technical means for providing the generated video so that the user can check it, and accepting correction instructions from the user as necessary.

[1042] "Means for providing the final video to the user" refers to the technical means for providing the final video approved by the user in a downloadable format or as a shareable link.

[1043] A "facial recognition system" is a technology that analyzes a user's facial image and identifies a specific emotional state.

[1044] "Voice input" is a technology that analyzes a user's voice, converts the content into text, and identifies the user's emotional state from the voice.

[1045] "Means for adjusting the content of explanatory scenarios and visual materials based on the user's emotions" refers to technical means for adaptively adjusting the tone and style of the scenarios and visual materials to be generated using the user's emotional data.

[1046] This invention is a system that uploads materials, analyzes their contents, generates videos, and provides them in the most appropriate format while recognizing the user's emotions. The program processing flow and operation of this system are described in detail below.

[1047] Uploading and Loading Materials

[1048] The user uploads the selected material (PDF, text file, etc.) to the system. They open a web browser on their device, select the material on the system's upload screen, and click the "Upload" button. This operation sends the material to the server. Furthermore, the emotion engine recognizes the user's emotional state (happiness, sadness, surprise, etc.) during uploading using a facial recognition system or voice input.

[1049] Text analysis of materials

[1050] The server receives and saves the uploaded file. It checks the file format and prepares the content for analysis using a PDF parser or text parser. The content of the document is then analyzed. For PDFs, a PDF parser (e.g., PyMuPDF or PDFMiner) is used to extract text data, and OCR technology (e.g., Tesseract OCR) is used to convert the characters in the image into text. For text files, the text data is read directly. Next, natural language processing (NLP) technology (e.g., spaCy or NLTK) is used to extract key information (keywords, important sentences, clauses, etc.). Based on the analyzed text data, the emotion engine adjusts the content output according to the user's emotional state (e.g., joy, sadness, surprise, etc.).

[1051] Scenario generation based on the content of the materials

[1052] The server generates a scenario for the commentary based on the analyzed text data. This scenario includes the flow of the commentary, narration content, key points, and insertion points for visual materials. Furthermore, an emotion engine evaluates the user's emotions and adjusts the content of the scenario accordingly. For example, if the user is having fun, the tone will be changed to a slightly more casual one, and if the user is sad, the explanation will be given in a warmer tone.

[1053] Voice synthesis and visual material creation

[1054] The server creates narration audio and visual materials based on the generated scenario. It uses text-to-speech (TTS) technology (e.g., Google Text-to-Speech or Amazon Polly) to convert text into speech and create the narration. At the same time, visual materials (e.g., illustrations, diagrams, etc.) are generated using image generation AI (e.g., DALL-E or MidJourney) or image editing tools (e.g., Photoshop). The emotion engine selects the voice tone and visual style (e.g., softer colors and designs) that match the user's emotional state.

[1055] Video Generation

[1056] The server generates a video by integrating the narration audio and visual materials. It uses a video editing engine (e.g., FFmpeg or OpenShot) to place the audio files and visual materials on a timeline. It also adds transition effects and text overlays to improve the visual quality of the video. Based on instructions from the emotion engine, it also adds effects and music that match the user's emotions.

[1057] Preview and edit your video

[1058] The server provides the generated video for the user to preview. The user then checks the video on their browser to see if the content matches their intentions. At this time, the emotion engine recognizes the user's emotions again and evaluates their reaction to the video content. If necessary, the user can specify areas to be corrected and provide feedback. The re-edited parts are adjusted taking into account the user's emotional state.

[1059] Download or share the video

[1060] The server delivers the final video that the user approves. The user can then download the video or generate a sharing link to share it with others. The server generates a secure download link so the user can save the video locally. It also generates a viewing link for internal or external sharing, and provides email and QR code generation functions as needed.

[1061] Examples of concrete examples and prompts

[1062] For example, consider a case where a parent creates an instructional video for their child about "constellations in space." When the parent uploads the material, the emotion engine recognizes the parent's emotions and detects that the parent is having fun. The server creates an instructional scenario with a fun, casual tone and selects bright colors and friendly visual materials. The parent can preview the generated video and make fine adjustments based on the parent's emotional response. After the corrections are reflected, the parent can download the final video and show it to their child.

[1063] Examples of prompts include:

[1064] "How can I upload a PDF file, detect the emotion of joy, and generate a casual explainer video based on that?"

[1065] "Please explain the steps for users to upload materials online, perform real-time emotion recognition, and use that data to create a video."

[1066] This system allows users to quickly convey accurate and easy-to-understand information in a video format that best suits their emotional state. Taking emotions into consideration leads to more effective communication.

[1067] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1068] Step 1:

[1069] The user uploads the material.

[1070] Input: User selected material (PDF, text file, etc.).

[1071] Operation: Open the device's web browser, select the file on the system's upload screen, and click the "Upload" button. Emotional data is also acquired through the user's facial recognition system and voice input.

[1072] Output: The selected material file is sent to the server, along with the user's emotion data.

[1073] Step 2:

[1074] The server receives the material.

[1075] Input: Uploaded data files, user emotion data.

[1076] Operation: The server saves the received file in the specified directory. It determines the file format (PDF, text file, etc.).

[1077] Output: The path of the saved document file, file format information.

[1078] Step 3:

[1079] The server analyzes the data.

[1080] Input: Saved material file, file format information.

[1081] How it works: For PDF files, the server extracts text using a PDF parser (e.g., PyMuPDF or PDFMiner) and converts characters in images to text using OCR technology (e.g., Tesseract OCR). For text files, the server reads the text data directly and extracts key information using natural language processing (NLP) technology (e.g., spaCy or NLTK).

[1082] Output: Parsed text data, extracted key information.

[1083] Step 4:

[1084] The emotion engine evaluates the user's emotions.

[1085] Input: User emotion data, parsed text data, extracted key information.

[1086] How it works: The emotion engine evaluates the analyzed text data based on the user's emotional state (happiness, sadness, surprise, etc.) and adjusts the content output.

[1087] Output: Emotion evaluation results, adjusted content output instructions.

[1088] Step 5:

[1089] The server generates the scenario.

[1090] Input: Parsed text data, extracted key information, sentiment evaluation results, and tailored content output instructions.

[1091] Operation: Based on these input data, the server generates a commentary scenario, which includes the flow of the commentary, narration content, important points, and insertion points for visual materials.

[1092] Output: The generated explanatory scenario.

[1093] Step 6:

[1094] The server creates the audio and visual material.

[1095] Input: Generated commentary scenario, emotion evaluation results.

[1096] How it works: The server uses voice synthesis technology (e.g., Google Text-to-Speech or Amazon Polly) to create the narration, and image generation AI (e.g., DALL-E or MidJourney) or image editing tools (e.g., Photoshop) to generate the visuals. The voice tone and visual style are adjusted to match the user's emotional state.

[1097] Output: Generated narration audio and visual materials.

[1098] Step 7:

[1099] The server generates the video.

[1100] Input: Generated narration audio, visual materials, and emotion evaluation results.

[1101] How it works: The server uses a video editing engine (e.g., FFmpeg or OpenShot) to combine audio and visual material into a single video file. It also adds transition effects and text overlays to enhance the visual quality. It also adds effects and music that suit the user's emotions.

[1102] Output: The generated video file.

[1103] Step 8:

[1104] The server provides a preview of the generated video.

[1105] Input: Generated video files, emotion evaluation results.

[1106] How it works: The server provides the user with a link to preview the video in their browser. The user reviews the video content and makes corrections as needed. The emotion engine again recognizes the user's emotions and evaluates their reactions during the preview.

[1107] Output: User's correction instructions, updated emotion evaluation results.

[1108] Step 9:

[1109] The server provides the final video.

[1110] Input: Generated video file, correction instructions from user.

[1111] How it works: The server re-edits the video based on the correction instructions and generates the final version. It then provides the final video to the user and generates a download link and a sharing link.

[1112] Output: Final video file, secure download link, and share link.

[1113] (Application example 2)

[1114] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1115] In conventional video generation systems, the process of analyzing the content of materials and creating videos is uniformly created without considering the user's emotional state, making it difficult to provide personalized content. Furthermore, in order for users to optimally understand and enjoy the content of videos, adjustments based on the user's emotions are necessary, but there has been a lack of systems that can achieve this.

[1116] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1117] In this invention, the server includes means for uploading materials, means for analyzing the content of the uploaded materials, means for generating an explanatory scenario based on the analyzed content, means for creating a narration voice and visual materials based on the generated scenario, means for generating a video by integrating the narration voice and visual materials, means for previewing the generated video and accepting correction instructions, means for recognizing the emotional state of the user and adjusting the scenario and visual style according to the emotion, and means for providing the user with the final generated video, thereby making it possible to provide optimal content according to the user's emotion.

[1118] "Means for uploading materials" refers to a device or method for uploading materials owned by a user into the system.

[1119] "Means for analyzing the content of uploaded material" means a device or method used to computerize and understand the content of uploaded material.

[1120] The "means for generating a commentary scenario" is a device or method for creating a plot or text for explanation or narration based on the analyzed content.

[1121] The "means for creating narration audio and visual materials" refers to a device or method for creating audio data and visual materials based on a generated scenario.

[1122] "Means for generating video" refers to a device or method for combining narration audio and visual materials to edit and generate a single video.

[1123] The "means for previewing the generated video and accepting correction instructions" refers to a device or method that allows a user to view the generated video in advance and indicate any necessary corrections.

[1124] "Means for recognizing the user's emotional state and adjusting the scenario and visual style according to that emotion" refers to a device or method that reads the user's emotions from their facial expressions, voice, etc., and changes or adjusts the content of the scenario and the visual design based on that.

[1125] The "means for providing the final video to the user" refers to a device or method that allows the user to download the completed video or share it with others.

[1126] An embodiment of the present invention will be described. This system analyzes materials uploaded by users and generates optimal explanatory videos while recognizing their emotional state. This system makes it possible to provide personalized video content that reflects the emotions of each individual user.

[1127] Uploading and Loading Materials

[1128] Users access the system's upload screen using a smartphone or computer browser. On the upload screen, users select and upload documents in PDF or text file format. At the same time as uploading, the system uses the device's built-in camera and microphone to recognize the user's emotional state through a facial recognition system or voice input. This emotion recognition uses facial expression recognition software and voice analysis software.

[1129] Text analysis of materials

[1130] The server receives the uploaded documents and extracts the text data using a PDF parser and OCR technology. This converts the text information from the images in the documents into text. Key information is then extracted using natural language processing (NLP). At this stage, the server also adjusts the analysis results based on the user's emotional state.

[1131] Scenario Generation

[1132] The server generates a commentary scenario for the user based on the emotional state and the analyzed text data. The generated scenario includes the flow of the commentary, the narration content, and the insertion points of visual materials. The emotion engine evaluates the user's emotions and adjusts the content of the scenario. For example, if the user is having fun, it will use casual language and select visually bright materials.

[1133] Creation of narration audio and visual materials

[1134] The server uses text-to-speech (TTS) technology to generate narration based on the generated scenario. At the same time, it creates visual materials such as illustrations and diagrams using generative AI models and image editing tools. At this stage, it also selects the style and tone that best suits the user's emotions.

[1135] Video Generation

[1136] The server combines the generated narration audio with the visual material and generates a video using a video editing engine (e.g., FFmpeg or OpenShot). The narration and visual material are placed on a timeline, and the visual quality is improved by adding transition effects and text overlays. Again, visual effects and music are applied based on the instructions of the emotion engine.

[1137] Preview and edit your video

[1138] The video generated by the server is previewed by the user. The user checks the video in their browser to ensure that the content is as intended. At the same time, the emotion engine evaluates the user's emotional response to the video. If necessary, the user can specify corrections and the video is re-edited.

[1139] Download or share the video

[1140] The final video is then delivered to the user via the server, who can either download the video or generate a shareable link to share it with others. The server generates a secure download link and also provides a viewing link for internal or external sharing.

[1141] Specific examples

[1142] For example, if a parent wants to create a learning video for their child on "Earth Geography," the system will automatically detect the parent's emotional state when they upload the materials. If a happy emotion is detected, a casual and fun-toned commentary scenario will be generated. Brightly colored diagrams and animations will be selected, and the parent can fine-tune the video while previewing it. Finally, a fun-toned geography learning video will be generated, which the parent can show to their child.

[1143] Prompt Sentence Examples

[1144] "Analyze this material and generate fun narration and colorful visuals to create a geography learning video for children."

[1145] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1146] Step 1:

[1147] Users access the system's upload screen using their devices, select and upload documents in PDF or text file format. The input data is the document file owned by the user, and the output data is the document file sent to the server. The user's emotional state is also collected at the same time using a facial recognition system and voice input. The input data is the user's facial expressions and voice, and the output data is the emotion recognition results.

[1148] Step 2:

[1149] The server analyzes the received document file. In the case of PDF, it uses a PDF parser to extract the text data and OCR technology to convert the characters in the image into text. The input data is the document file sent to the server, and the output data is the extracted text data.

[1150] Step 3:

[1151] The server uses natural language processing (NLP) techniques to extract keywords and important sentences from the extracted text data. The input data is the extracted text data, and the output data is a list of keywords and important sentences.

[1152] Step 4:

[1153] The server generates an explanatory scenario based on the emotion recognition results and the analyzed text data. The input data is a list of keywords and important sentences and the emotion recognition results, and the output data is the generated explanatory scenario. The emotion recognition results affect the tone and content of the scenario. For example, a casual scenario is generated for a positive emotional state.

[1154] Step 5:

[1155] The server uses text-to-speech (TTS) technology to create a narration voice based on the generated scenario. The input data is the generated commentary scenario, and the output data is a narration voice file. The TTS engine converts the text into speech.

[1156] Step 6:

[1157] The server creates visual materials using generative AI models and image editing tools. The input data is the generated explanatory scenario, and the output data is a set of visual materials. Diagrams and illustrations are generated to match the key points of the scenario.

[1158] Step 7:

[1159] The server uses a video editing engine (e.g., FFmpeg or OpenShot) to combine the narration audio and visual materials to generate a video. The input data is a narration audio file and a set of visual materials, and the output data is the generated video file. Transition effects and text overlays are also added.

[1160] Step 8:

[1161] The server allows the user to preview the generated video. The user checks the video on the browser and provides feedback on any corrections. The input data is the generated video file, and the output data is the user's feedback. The emotion engine reevaluates the user's emotions about the video and identifies areas that need to be re-edited.

[1162] Step 9:

[1163] The server re-edits the video based on the user's feedback and makes final adjustments according to the emotions. The input data are the user's feedback and the emotion re-evaluation results, and the output data is the final video file.

[1164] Step 10:

[1165] The server provides the final video to the user, who can then download it or share it with others via a sharing link. The input data is the final video file, and the output data is a download link and a sharing link. The server generates a secure download link and provides a viewing link.

[1166] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1167] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1168] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1169] [Fourth embodiment]

[1170] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1171] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1172] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1173] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1174] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1175] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1176] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1177] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1178] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1179] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1180] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1181] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1182] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1183] This invention is a system that uploads materials, analyzes the contents, generates videos, and provides them to users. The specific program processing flow and system operation are described in detail below.

[1184] Uploading and Loading Materials

[1185] The user uploads the selected book, dictionary, picture book, or material (PDF, text file, etc.) to the system. Using the device's web browser, the user selects the material on the designated upload screen and clicks the upload button. This operation sends the material to the server.

[1186] Text analysis of materials

[1187] The server analyzes the uploaded document. It uses a PDF parser and text parser to extract the text data from the document. It then uses natural language processing (NLP) technology to extract keywords and important sentences from the text and analyze the structure of the document. If necessary, OCR (optical character recognition) technology is used to convert text in images into text.

[1188] Scenario generation based on the content of the materials

[1189] The server generates a commentary scenario based on the analyzed text data. This is the process of determining the flow of the video and defining the narration content and insertion points for necessary visual materials. Structured data is created, specifying the commentary order and detailed narration content for each section.

[1190] Voice synthesis and visual material creation

[1191] The server creates narration audio and visual materials based on the generated scenario. It uses Text-to-Speech (TTS) technology to convert text into audio and create the narration. At the same time, visual materials (illustrations, diagrams, etc.) are generated using image generation AI and image editing tools.

[1192] Video Generation

[1193] The server combines the narration audio with the visual material to generate the video. Using a video editing engine (such as FFmpeg or OpenShot), the material is placed on a timeline and transition effects and text overlays are added to create a visually appealing video. At this stage, the video is exported to its final form and saved in an appropriate format, such as MP4.

[1194] Preview and edit your video

[1195] The user can preview the generated video and check it on the browser. The server provides a preview function that enables this, and after the user plays the video and checks the content, they can provide feedback on any necessary corrections. Upon receiving the correction request, the server re-edits the relevant parts and updates the video.

[1196] Download or share the video

[1197] Users can download the final video or generate a sharing link to share it with others. The server generates a secure download link, allows users to save the video locally, and generates a viewing link for internal or external sharing, and also provides the ability to send an email or generate a QR code if desired.

[1198] As a concrete example, let's consider the case where a parent creates an explanatory video for their child about the planets of the solar system. The parent uploads a PDF of an elementary school science textbook and inputs it into the system. The server analyzes the material and extracts information about the planets in the solar system. An explanatory scenario is then generated, and narration audio and planet illustrations are created. These materials are then integrated into a video. The parent can use the preview function to check the video and make any final corrections. After the corrections are complete, the parent can download the final video and show it to their child.

[1199] Through this series of steps, users can easily provide accurate information in video format, saving a great deal of time and effort. By using this system, information can be transmitted quickly and effectively.

[1200] The processing flow will be explained below.

[1201] Step 1:

[1202] The user uploads the selected material (PDF, text file, etc.) to the system by opening a web browser on the terminal, selecting the material on the system's upload screen, and clicking the upload button.

[1203] Step 2:

[1204] The server receives and stores the uploaded file, first checking the file format and preparing to parse the content using a PDF parser or text parser.

[1205] Step 3:

[1206] The server analyzes the content of the document. In the case of a PDF, it uses a PDF parser to extract the text data and OCR technology to convert the characters in the image into text. In the case of a text file, it reads the text data directly. Next, it uses natural language processing (NLP) technology to extract key information (keywords, important sentences, paragraphs, etc.).

[1207] Step 4:

[1208] The server generates a commentary scenario based on the analyzed text data. The scenario includes the flow of the commentary, narration content, important points, and points where visual materials should be inserted. The scenario is organized as structured data.

[1209] Step 5:

[1210] The server creates narration audio and visual materials based on the scenario. It uses text-to-speech (TTS) technology to convert the scenario text into narration audio. At the same time, it uses image generation AI and image editing tools to generate the necessary visual materials, such as illustrations and diagrams.

[1211] Step 6:

[1212] The server generates a video by integrating the narration audio and visual materials. Using a video editing engine (e.g., FFmpeg or OpenShot), the audio files and visual materials are placed on a timeline. Furthermore, transition effects and text overlays are added to improve the visual quality of the video.

[1213] Step 7:

[1214] The server provides the generated video so that the user can preview it. The user can check the video in the browser and check whether the content matches the user's intention. If necessary, the user can specify corrections and provide feedback.

[1215] Step 8:

[1216] The server receives correction instructions from the user, re-edits the relevant parts, and provides the re-edited video for preview again so that the user can make a final check.

[1217] Step 9:

[1218] The server then provides the final video that the user has approved. The user can then download the video or generate a sharing link to share it with others. The server generates secure download links and viewing links, and also provides email and QR code generation functions as needed.

[1219] By following the above steps, users can easily create and quickly provide accurate and easy-to-understand instructional videos based on materials of their own choosing. This system can be used for a wide range of purposes, such as home education and creating business manuals in companies.

[1220] Example 1

[1221] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1222] Conventional document analysis systems often lack sufficient methods for efficiently understanding the content of documents. In particular, it is difficult to quickly and accurately analyze vast amounts of data and provide the information in a format that is easy for users to understand, which requires time and effort. Furthermore, simply providing the content of documents in text format makes it difficult to convert it into a visually understandable format, making it ineffective for use in education, presentations, and other applications. Therefore, there is a growing demand for a system that can analyze uploaded documents and provide information in video format based on the results.

[1223] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1224] In this invention, the server includes means for uploading materials, means for analyzing the content of the uploaded materials, means for generating an explanatory scenario based on the analyzed content, means for creating a narration voice and visual materials based on the generated scenario, means for generating a video by integrating the narration voice and visual materials, means for previewing the generated video and accepting correction instructions, and means for providing the final generated video to the user. This makes it possible to analyze the content of the uploaded materials and provide the information quickly and efficiently in a video format that is easy for the user to understand.

[1225] The "means for uploading materials" is a function for transmitting materials selected by the user to the system and storing them on the server.

[1226] "Means for analyzing the content of uploaded materials" refers to a function for extracting text data and other information from uploaded materials and converting that information into an easily understandable format.

[1227] The "means for generating an explanatory scenario" is a function that creates a scenario for determining the structure of the explanatory video and the content of the narration based on the content of the analyzed materials.

[1228] "Means for creating narration audio and visual materials" refers to a function for creating narration audio and visual materials using voice synthesis technology and image generation technology in accordance with the generated scenario.

[1229] "Means for integrating narration audio and visual materials to generate video" refers to a function that combines the created audio and visual materials on a timeline and outputs them as a completed video.

[1230] The "means for previewing the generated video and accepting correction instructions" is a function that allows the user to check the generated video and provide feedback on parts that need correction.

[1231] The "means for providing the final video to the user" is a function that allows the user to download the finalized video or generate a link that allows the user to share it with other people.

[1232] "Natural language processing technology" is a method for analyzing text data, extracting keywords and important sentences from it, and understanding the content of the material.

[1233] "Speech synthesis technology" is a technology that generates synthetic speech based on text data.

[1234] "Image generation technology" refers to the technology of creating visual materials based on text or other data.

[1235] A "video editing engine" is software or a system for generating and editing videos by combining narration audio and visual materials.

[1236] The present invention provides a system for uploading materials, analyzing the contents of the materials, generating videos, and providing the videos to users. Specific embodiments will be described in detail below.

[1237] Uploading and Loading Materials

[1238] The user accesses the system's upload screen using a web browser on their device and uploads the selected materials (PDF, text files, etc.). The user clicks the upload button, which sends the materials to the server.

[1239] Analysis of data

[1240] The server receives the uploaded documents and temporarily stores them in storage. It then uses a PDF parser (pdfminer.six) to extract text data from the PDF file. For text files, it directly reads the content. It then uses natural language processing (NLP) technology (NLTK) to extract keywords and important sentences from the text and analyze the structure of the document. If necessary, it also converts text within images into text using OCR (optical character recognition) technology (Tesseract OCR).

[1241] Generate a scenario

[1242] The server generates an explanatory scenario based on the analyzed text data. This scenario includes the flow of the video, the narration content, and insertion points for visual materials. To generate the scenario, a generative AI model is used and a prompt is entered. For example, the prompt might read, "Please create a scenario that explains the planets in the solar system for elementary school science classes. I have uploaded a PDF of a science textbook as a reference. Based on this reference, please generate a scenario that includes the characteristics, relative positions, and important information about each planet."

[1243] Voice synthesis and visual material creation

[1244] Based on the generated scenario, the server uses text-to-speech (TTS) technology (Google Text-to-Speech API) to convert text into speech and create the narration. At the same time, it uses image generation AI (e.g., DALL-E) and image editing tools (Adobe Photoshop) to create the necessary visual materials.

[1245] Video generation

[1246] The server integrates the created narration audio with the visual material, and then uses a video editing engine (e.g., FFmpeg) to place the material on a timeline, adding transition effects and text overlays to create a visually appealing video.

[1247] Preview and edit your video

[1248] The user can preview the generated video on the browser and check the content. The server provides a preview function and gives feedback to the user on any corrections that need to be made. The server receives the correction request, re-edits the relevant parts, and updates the video.

[1249] Video provision

[1250] Users can then download the final video or share it with others. The server generates a secure download link and provides it to the user. It can also generate a sharing link for viewing, which can be sent via email or generated as a QR code.

[1251] As a specific example, if a parent wants to create an explanatory video for their child about the planets in the solar system, they upload a PDF of a science textbook to the system. The server analyzes the material and extracts information about the planets in the solar system. The generative AI model then generates a scenario and uses speech synthesis and image generation technologies to create narration audio and illustrations of the planets. These are then integrated to generate a video, and the user (parent) can use the preview function to check the content, make any necessary corrections, and then download or share the final video.

[1252] This system allows users to easily create high-quality videos for educational and presentation purposes.

[1253] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1254] Step 1:

[1255] The user accesses the system's upload screen using the terminal's web browser, selects the document file (PDF, text file, etc.), and clicks the upload button.

[1256] Input: Document file

[1257] Output: The document file is sent to the server.

[1258] Operation: When the user operates the application, the document file is sent to the server via an HTTP request and temporarily stored in storage.

[1259] Step 2:

[1260] The server analyzes the received data file.

[1261] Input: Uploaded file

[1262] Output: Parsed text data

[1263] Operation: For PDF files, it uses the PDF parser (pdfminer.six) to extract text data. For text files, it reads the content directly. If necessary, it also uses OCR (Tesseract OCR) technology to convert characters in images into text.

[1264] Step 3:

[1265] The server uses natural language processing (NLP) technology (NLTK) to extract keywords and important sentences from the analyzed text data.

[1266] Input: Parsed text data

[1267] Output: Extracted keywords and important sentences

[1268] How it works: NLP technology analyzes text and converts the structure of the material into an easy-to-understand format.

[1269] Step 4:

[1270] Based on the extracted information, the server uses a generative AI model to generate an explanatory scenario.

[1271] Input: Extracted keywords and important sentences

[1272] Output: Explaining scenario

[1273] How it works: A prompt for scenario generation is input into the generative AI model, which then creates a scenario that includes the flow of explanations, narration content, and insertion points for visual materials. For example, the prompt might be, "Please create a scenario that explains the planets in the solar system for use in elementary school science classes."

[1274] Step 5:

[1275] The server creates narration audio based on the generated scenario using a speech synthesis API (Google Text-to-Speech API).

[1276] Input: Explaining scenario

[1277] Output: Narration audio file

[1278] Operation: The scenario text is input into the speech synthesis API, synthetic speech is generated, and saved as a narration audio file.

[1279] Step 6:

[1280] Based on the generated scenario, the server creates visual materials using image generation AI and image editing tools.

[1281] Input: Explaining scenario

[1282] Output: Visual material (image file)

[1283] How it works: Using an image generation AI (e.g., DALL-E) or an image editing tool (Adobe Photoshop), the necessary illustrations and diagrams are generated and saved as image files.

[1284] Step 7:

[1285] The server combines the narration audio and visual material to generate a video.

[1286] Input: Narration audio file, visual material

[1287] Output: Generated video file

[1288] How it works: Use a video editing engine (e.g. FFmpeg) to combine audio and visual elements on the timeline, add transition effects and text overlays to create a video, and save the finished video in an appropriate format, such as MP4.

[1289] Step 8:

[1290] The user can preview the generated animation and provide any necessary corrections.

[1291] Input: Generated video file

[1292] Output: Correction instructions

[1293] How it works: The user plays the video and provides feedback using the preview functionality provided by the server. The server receives correction requests, re-edits the relevant parts, and updates the video.

[1294] Step 9:

[1295] Users can then download the final edited video or generate a shareable link to share it with others.

[1296] Input: Final generated video

[1297] Output: Download link or share link

[1298] How it works: The server generates a secure download link and provides it to the user. The user can then download the video or share it with others using a sharing link. The server also provides the ability to email the sharing link or generate a QR code.

[1299] (Application example 1)

[1300] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1301] Logistics centers need a method to efficiently and quickly communicate work procedures and equipment operation methods to workers. Traditionally, manuals using text and still images have been widely used, but they have problems such as being difficult to understand and being difficult to update. In addition, when new work procedures or equipment are introduced, training is required, which takes time and effort. To solve these problems, a more effective and easy-to-understand method is needed.

[1302] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1303] In this invention, the server includes means for uploading materials, means for analyzing the contents of the uploaded materials, means for generating an explanatory scenario based on the analyzed contents, means for creating a narration voice and visual materials based on the generated scenario, means for generating a video by integrating the narration voice and visual materials, means for customizing the generated video for the logistics center and displaying it on a smart terminal or a wearable device so that workers can check it, means for previewing the generated video and accepting correction instructions, and means for providing the final generated video to the user. This makes it possible to provide work procedures and equipment operation methods in a video format that is instantly easy to understand.

[1304] "Documents" are document data uploaded by users, such as PDFs or text files.

[1305] "Uploading means" refers to a means that has the function of allowing a user to transfer materials to a server via a network.

[1306] "Means for analyzing" means means including technology for analyzing the content of uploaded materials and converting them into an understandable form.

[1307] An "explanatory scenario" is a plan that defines the flow and sequence of video and audio, generated based on the content of the analyzed materials.

[1308] "Narration voice" is voice data created based on a commentary scenario, and conveys the commentary content through sound.

[1309] "Visual materials" are visual materials such as illustrations and diagrams used to complement the explanatory scenario.

[1310] "Means for generating moving images" means means including technology for integrating narration audio and visual materials into a single moving image format.

[1311] "Customizing for the distribution center" means adjusting the format and content of the generated video to be suitable for use in the distribution center.

[1312] "Smart devices" refer to portable information terminals such as smartphones and tablets.

[1313] A "wearable device" is a device that can be worn by a worker and that displays information.

[1314] "Preview" means temporarily viewing the generated video for final confirmation.

[1315] A "correction instruction" is a request for correction or modification made by a user to a generated video.

[1316] The present invention is a system for automatically generating training videos for a logistics center, which operates mainly on a server and a user terminal. A specific embodiment of the system is described below.

[1317] System configuration

[1318] The system consists of the following main components:

[1319] 1. User devices: Including smartphones, PCs, smart devices, wearable devices, etc. Data is uploaded from here.

[1320] 2. Server: The main component that analyzes materials, generates scenarios, synthesizes voice, generates visual materials, and generates videos.

[1321] 3. Network: Responsible for data communication between user terminals and servers.

[1322] Key Features

[1323] Upload materials:

[1324] Materials (PDF or text files) are uploaded to the server from the user's device. Users select and upload materials using a browser-based interface.

[1325] Content analysis of the material:

[1326] The server analyzes the uploaded documents, extracting text data using a PDF parser or text parser, and extracting important keywords and sentences using natural language processing (NLP). It also converts text in images into text using OCR technology, if necessary.

[1327] Generate an explanatory scenario:

[1328] The server generates a commentary scenario based on the analyzed text data, which determines the flow of the video and defines the insertion points for the narration content and necessary visual materials.

[1329] Narration audio and visual material creation:

[1330] The server uses Text-to-Speech (TTS) technology to create narration based on the generated scenario, and generates visual materials (illustrations, diagrams, etc.) using image generation AI and image editing tools.

[1331] Video Generation:

[1332] The server generates a video by integrating the narration audio and visual materials. Using a video editing engine (such as FFmpeg or OpenShot), the materials are placed on a timeline and transition effects and text overlays are added to create a visually appealing video.

[1333] Customize and view the generated video:

[1334] The generated videos are customized for the logistics center and displayed on smart terminals and wearable devices, allowing workers to easily review the videos and apply them to their work.

[1335] Preview and fix:

[1336] The user can preview the generated video and check it in their browser. The server provides a preview function that allows the user to play the video, check the content, and then provide feedback on any necessary corrections.

[1337] Final video credit:

[1338] Users can download the final video or generate a share link to share it with others. The server generates a secure download link, allowing users to save the video locally.

[1339] Specific examples

[1340] For example, if you upload an operating procedure manual (PDF) for a newly introduced packaging machine, the system will analyze the contents and generate a step-by-step scenario like the one below.

[1341] "Step 1: Power on the machine."

[1342] "Step 2: Select the appropriate settings in the panel."

[1343] "Step 3: Place the item in the machine."

[1344] "Step 4: Press the Start button."...and so on.

[1345] The training video is then created by synthesizing the voice and inserting images and diagrams corresponding to each step.

[1346] Prompt Sentence Examples

[1347] Below are prompts for inputting new information into a generative AI model.

[1348] Analyze the following procedure, generate narration text that clearly explains each step, and generate a scenario for creating a training video based on that.

[1349] Instructions:

[1350] Turn on the machine.

[1351] Select the appropriate settings in the panel.

[1352] Place the product in the machine.

[1353] Press the start button.

[1354] Make sure the product is packaged.

[1355] Stop the machine.

[1356] By following the above steps, it is possible to efficiently generate training videos for logistics centers.

[1357] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1358] Step 1:

[1359] This is the process of uploading materials from a user's device. The user uses a browser-based interface to select materials such as PDF or text files and clicks the upload button. The input is the file specified by the user, and the output is the file transfer to the server. This operation sends the selected materials to the server.

[1360] Step 2:

[1361] This is the process of analyzing the content of documents on the server. After the server receives the uploaded documents, it first extracts the text data using a PDF parser or text parser. The input is the uploaded document, and the output is the extracted text data. At the same time, OCR technology is used to extract the text information from the image, thereby extracting pure text information.

[1362] Step 3:

[1363] This is a process that generates an explanatory scenario based on the text data analyzed by the server. The server uses natural language processing (NLP) technology to extract keywords and important sentences from the text and analyze the structure of the material. The input is the extracted text data, and the output is an explanatory scenario. The scenario defines the flow of the video.

[1364] Step 4:

[1365] This is a process that creates narration audio based on a scenario generated on the server. The server uses Text-to-Speech (TTS) technology to convert the scenario text into audio. The input is the explanatory scenario, and the output is the narration audio file. This generates audio data corresponding to the scenario.

[1366] Step 5:

[1367] At the same time, the server creates visual materials. Image generation AI and image editing tools are used to create the necessary illustrations and diagrams based on the explanatory scenario. The input is the explanatory scenario, and the output is the visual materials. This generates image data to visually complement the data.

[1368] Step 6:

[1369] This is the process of integrating narration audio and visual materials on the server to generate a video. Using a video editing engine (such as FFmpeg or OpenShot), the materials are placed on a timeline and transition effects and text overlays are added. The input is the narration audio and visual materials, and the output is the generated video file.

[1370] Step 7:

[1371] This process customizes the generated video for the logistics center and displays it on a smart terminal or wearable device so that workers can review it. The input is the generated video file, and the output is the video displayed on a smart terminal or wearable device. This allows workers to learn their work while watching the video on-site.

[1372] Step 8:

[1373] This process previews the generated video and accepts correction instructions. The user plays the video in a browser and provides feedback on any necessary corrections. The input is the generated video file, and the output is correction instructions from the user.

[1374] Step 9:

[1375] This is the process of providing the final generated video to the user. The server generates a secure download link so that the user can save the video locally, and also generates a sharing link so that the video can be shared with other people. The input is the generated video file, and the output is the download link or sharing link.

[1376] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1377] This invention is a system that uploads materials, analyzes their contents, generates videos, and provides them in an optimal format while recognizing the user's emotions. The specific program processing flow and system operation are described in detail below.

[1378] Uploading and Loading Materials

[1379] The user uploads the selected material (PDF, text file, etc.) to the system. They open a web browser on their device, select the material on the system's upload screen, and click the upload button. This operation sends the material to the server. Furthermore, the emotion engine recognizes the user's emotional state (happiness, sadness, surprise, etc.) during uploading using a facial recognition system or voice input.

[1380] Text analysis of materials

[1381] The server receives and saves the uploaded file. It checks the file format and prepares to analyze the content of the document using a PDF parser or text parser. The content of the document is then analyzed. In the case of a PDF, the PDF parser is used to extract text data, and OCR technology is used to convert the characters in the image into text. In the case of a text file, the text data is read directly. Natural language processing (NLP) technology is then used to extract key information (keywords, important sentences, paragraphs, etc.). Based on the analyzed text data, the emotion engine adjusts the content output according to the user's emotional state (e.g., joy, sadness, surprise, etc.).

[1382] Scenario generation based on the content of the materials

[1383] The server generates a scenario for the commentary based on the analyzed text data. This scenario includes the flow of the commentary, narration content, key points, and insertion points for visual materials. Furthermore, an emotion engine evaluates the user's emotions and adjusts the content of the scenario accordingly. For example, if the user is having fun, the tone will be changed to a slightly more casual one, and if the user is sad, the explanation will be given in a warmer tone.

[1384] Voice synthesis and visual material creation

[1385] The server creates narration audio and visual materials based on the generated scenario. Text-to-speech (TTS) technology is used to convert text into audio and create the narration. At the same time, visual materials (illustrations, diagrams, etc.) are generated using image generation AI and image editing tools. The emotion engine selects the audio tone and visual style (e.g., softer colors and designs) that match the user's emotional state.

[1386] Video Generation

[1387] The server generates a video by integrating the narration audio and visual materials. It then uses a video editing engine (e.g., FFmpeg or OpenShot) to place the audio files and visual materials on a timeline. It also adds transition effects and text overlays to improve the visual quality of the video. Based on instructions from the emotion engine, it also adds effects and music that match the user's emotions.

[1388] Preview and edit your video

[1389] The server provides the generated video for the user to preview. The user then checks the video on their browser to see if the content matches their intentions. At this time, the emotion engine recognizes the user's emotions again and evaluates their reaction to the video content. If necessary, the user can specify areas to be corrected and provide feedback. The re-edited parts are adjusted taking into account the user's emotional state.

[1390] Download or share the video

[1391] The server delivers the final video that the user approves. The user can then download the video or generate a sharing link to share it with others. The server generates a secure download link so the user can save the video locally. It also generates a viewing link for internal or external sharing, and provides email and QR code generation functions as needed.

[1392] As a concrete example, consider the case where a parent creates an instructional video for their child about "constellations in space." When the parent uploads the material, the emotion engine recognizes the parent's emotions and detects that the parent is having fun. The server creates an instructional scenario with a fun, casual tone and selects bright colors and friendly visual materials. The parent previews the generated video and makes fine adjustments based on the parent's emotional response. After the corrections are reflected, the parent can download the final video and show it to their child.

[1393] This system allows users to quickly convey accurate and easy-to-understand information in a video format that best suits their emotional state. Taking emotions into consideration leads to more effective communication.

[1394] The processing flow will be explained below.

[1395] Step 1:

[1396] The user uploads the selected material (PDF, text file, etc.) to the system. They open a web browser on their device, select the material on the system's upload screen, and click the upload button. At this time, the emotion engine recognizes the user's emotional state from their facial expressions and voice.

[1397] Step 2:

[1398] The server receives and stores the uploaded file, checks the file format, and prepares the content for analysis using a PDF parser or text parser.

[1399] Step 3:

[1400] The server analyzes the content of the document. In the case of a PDF, it uses a PDF parser to extract text data and OCR technology to convert the characters in the image into text. In the case of a text file, it reads the text data directly. Next, it uses natural language processing (NLP) technology to extract key information (keywords, important sentences, paragraphs, etc.).

[1401] Step 4:

[1402] The server generates explanatory scenarios based on the analyzed text data and the user's emotional state. For example, if the user looks happy, it generates a scenario with a casual and cheerful tone, and if the user looks sad, it generates a scenario with a gentler and warmer tone.

[1403] Step 5:

[1404] The server creates narration audio and visual materials based on the scenario. It uses text-to-speech (TTS) technology to convert the scenario text into narration audio. At the same time, it follows instructions from the emotion engine to create visual materials with designs and colors that match the user's emotional state. For example, bright colors are used for cheerful emotions, and softer colors are used for calm emotions.

[1405] Step 6:

[1406] The server generates a video by integrating the narration audio and visual materials. A video editing engine (e.g., FFmpeg or OpenShot) is used to place the audio files and visual materials on a timeline. Furthermore, transition effects and text overlays are added to improve the visual quality of the video. The emotion engine also selects effects and background music that match the user's emotions.

[1407] Step 7:

[1408] The server provides the generated video for the user to preview. The user can then view the video in their browser and check whether the content matches their intent. At this time, the emotion engine again recognizes the user's emotions and evaluates their reaction to the video.

[1409] Step 8:

[1410] The user provides feedback on the parts they want to edit, and the server receives the instructions. The emotion engine re-evaluates the user's emotional state, makes any necessary adjustments, and re-edits the video. The re-edited video is then provided for preview again, and the user makes a final confirmation.

[1411] Step 9:

[1412] The server delivers the final video that the user approves. The user can then download the video or generate a sharing link to share it with others. The server generates a secure download link so the user can save the video locally. It also generates a viewing link for internal or external sharing, and provides email and QR code generation functions as needed.

[1413] This series of steps allows users to easily create and quickly deliver accurate, easy-to-understand explainer videos that best suit their emotional state, making information delivery more effective and personalized.

[1414] Example 2

[1415] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1416] In recent years, the demand for digital content has increased, and there is a particular need for the generation of video content that is optimized for individual users' emotions. However, conventional systems have had difficulty providing adaptive video content that takes into account user emotions throughout the entire process: uploading materials, analyzing content, generating scenarios, creating narration audio, creating visual materials, generating videos, and previewing and editing. This has resulted in a poor user experience and made it difficult to achieve effective communication.

[1417] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1418] In this invention, the server includes means for uploading materials, means for analyzing the content of the uploaded materials, means for generating an explanatory scenario based on the analyzed content, means for creating a narration voice and visual materials based on the generated scenario, means for generating a video by integrating the narration voice and visual materials, means for previewing the generated video and accepting correction instructions, means for providing the final generated video to a user, means for recognizing the user's emotions using a face recognition system or voice input when uploading and previewing the materials, and means for adjusting the content of the explanatory scenario and visual materials based on the user's emotions, thereby enabling the rapid and effective creation of video content optimized for the user's emotional state.

[1419] "Means for uploading materials" refers to the technical means that provides an interface for users to send materials they possess to the system and store them on the server.

[1420] "Means for analyzing the content of uploaded materials" refers to the technical means by which the server extracts and understands the content of the materials received by the server using text analysis and image processing technology.

[1421] The "means for generating an explanatory scenario" is a technical means for automatically creating a scenario including explanations, narration flow, important points, etc. based on the analyzed content.

[1422] "Means for creating narration audio and visual materials" refers to the technical means for generating narration audio and visual materials using voice synthesis technology and image generation technology based on the generated scenario.

[1423] "Means for generating a video by integrating narration audio and visual materials" refers to means for combining narration audio and visual materials into a single video file, and including video editing technology for improving editing quality.

[1424] The "means for previewing the video and accepting correction instructions" refers to a technical means for providing the generated video so that the user can check it, and accepting correction instructions from the user as necessary.

[1425] "Means for providing the final video to the user" refers to the technical means for providing the final video approved by the user in a downloadable format or as a shareable link.

[1426] A "facial recognition system" is a technology that analyzes a user's facial image and identifies a specific emotional state.

[1427] "Voice input" is a technology that analyzes a user's voice, converts the content into text, and identifies the user's emotional state from the voice.

[1428] "Means for adjusting the content of explanatory scenarios and visual materials based on the user's emotions" refers to technical means for adaptively adjusting the tone and style of the scenarios and visual materials to be generated using the user's emotional data.

[1429] This invention is a system that uploads materials, analyzes their contents, generates videos, and provides them in the most appropriate format while recognizing the user's emotions. The program processing flow and operation of this system are described in detail below.

[1430] Uploading and Loading Materials

[1431] The user uploads the selected material (PDF, text file, etc.) to the system. They open a web browser on their device, select the material on the system's upload screen, and click the "Upload" button. This operation sends the material to the server. Furthermore, the emotion engine recognizes the user's emotional state (happiness, sadness, surprise, etc.) during uploading using a facial recognition system or voice input.

[1432] Text analysis of materials

[1433] The server receives and saves the uploaded file. It checks the file format and prepares the content for analysis using a PDF parser or text parser. The content of the document is then analyzed. For PDFs, a PDF parser (e.g., PyMuPDF or PDFMiner) is used to extract text data, and OCR technology (e.g., Tesseract OCR) is used to convert the characters in the image into text. For text files, the text data is read directly. Next, natural language processing (NLP) technology (e.g., spaCy or NLTK) is used to extract key information (keywords, important sentences, clauses, etc.). Based on the analyzed text data, the emotion engine adjusts the content output according to the user's emotional state (e.g., joy, sadness, surprise, etc.).

[1434] Scenario generation based on the content of the materials

[1435] The server generates a scenario for the commentary based on the analyzed text data. This scenario includes the flow of the commentary, narration content, key points, and insertion points for visual materials. Furthermore, an emotion engine evaluates the user's emotions and adjusts the content of the scenario accordingly. For example, if the user is having fun, the tone will be changed to a slightly more casual one, and if the user is sad, the explanation will be given in a warmer tone.

[1436] Voice synthesis and visual material creation

[1437] The server creates narration audio and visual materials based on the generated scenario. It uses text-to-speech (TTS) technology (e.g., Google Text-to-Speech or Amazon Polly) to convert text into speech and create the narration. At the same time, visual materials (e.g., illustrations, diagrams, etc.) are generated using image generation AI (e.g., DALL-E or MidJourney) or image editing tools (e.g., Photoshop). The emotion engine selects the voice tone and visual style (e.g., softer colors and designs) that match the user's emotional state.

[1438] Video Generation

[1439] The server generates a video by integrating the narration audio and visual materials. It uses a video editing engine (e.g., FFmpeg or OpenShot) to place the audio files and visual materials on a timeline. It also adds transition effects and text overlays to improve the visual quality of the video. Based on instructions from the emotion engine, it also adds effects and music that match the user's emotions.

[1440] Preview and edit your video

[1441] The server provides the generated video for the user to preview. The user then checks the video on their browser to see if the content matches their intentions. At this time, the emotion engine recognizes the user's emotions again and evaluates their reaction to the video content. If necessary, the user can specify areas to be corrected and provide feedback. The re-edited parts are adjusted taking into account the user's emotional state.

[1442] Download or share the video

[1443] The server delivers the final video that the user approves. The user can then download the video or generate a sharing link to share it with others. The server generates a secure download link so the user can save the video locally. It also generates a viewing link for internal or external sharing, and provides email and QR code generation functions as needed.

[1444] Examples of concrete examples and prompts

[1445] For example, consider a case where a parent creates an instructional video for their child about "constellations in space." When the parent uploads the material, the emotion engine recognizes the parent's emotions and detects that the parent is having fun. The server creates an instructional scenario with a fun, casual tone and selects bright colors and friendly visual materials. The parent can preview the generated video and make fine adjustments based on the parent's emotional response. After the corrections are reflected, the parent can download the final video and show it to their child.

[1446] Examples of prompts include:

[1447] "How can I upload a PDF file, detect the emotion of joy, and generate a casual explainer video based on that?"

[1448] "Please explain the steps for users to upload materials online, perform real-time emotion recognition, and use that data to create a video."

[1449] This system allows users to quickly convey accurate and easy-to-understand information in a video format that best suits their emotional state. Taking emotions into consideration leads to more effective communication.

[1450] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1451] Step 1:

[1452] The user uploads the material.

[1453] Input: User selected material (PDF, text file, etc.).

[1454] Operation: Open the device's web browser, select the file on the system's upload screen, and click the "Upload" button. Emotional data is also acquired through the user's facial recognition system and voice input.

[1455] Output: The selected material file is sent to the server, along with the user's emotion data.

[1456] Step 2:

[1457] The server receives the material.

[1458] Input: Uploaded data files, user emotion data.

[1459] Operation: The server saves the received file in the specified directory. It determines the file format (PDF, text file, etc.).

[1460] Output: The path of the saved document file, file format information.

[1461] Step 3:

[1462] The server analyzes the data.

[1463] Input: Saved material file, file format information.

[1464] How it works: For PDF files, the server extracts text using a PDF parser (e.g., PyMuPDF or PDFMiner) and converts characters in images to text using OCR technology (e.g., Tesseract OCR). For text files, the server reads the text data directly and extracts key information using natural language processing (NLP) technology (e.g., spaCy or NLTK).

[1465] Output: Parsed text data, extracted key information.

[1466] Step 4:

[1467] The emotion engine evaluates the user's emotions.

[1468] Input: User emotion data, parsed text data, extracted key information.

[1469] How it works: The emotion engine evaluates the analyzed text data based on the user's emotional state (happiness, sadness, surprise, etc.) and adjusts the content output.

[1470] Output: Emotion evaluation results, adjusted content output instructions.

[1471] Step 5:

[1472] The server generates the scenario.

[1473] Input: Parsed text data, extracted key information, sentiment evaluation results, and tailored content output instructions.

[1474] Operation: Based on these input data, the server generates a commentary scenario, which includes the flow of the commentary, narration content, important points, and insertion points for visual materials.

[1475] Output: The generated explanatory scenario.

[1476] Step 6:

[1477] The server creates the audio and visual material.

[1478] Input: Generated commentary scenario, emotion evaluation results.

[1479] How it works: The server uses voice synthesis technology (e.g., Google Text-to-Speech or Amazon Polly) to create the narration, and image generation AI (e.g., DALL-E or MidJourney) or image editing tools (e.g., Photoshop) to generate the visuals. The voice tone and visual style are adjusted to match the user's emotional state.

[1480] Output: Generated narration audio and visual materials.

[1481] Step 7:

[1482] The server generates the video.

[1483] Input: Generated narration audio, visual materials, and emotion evaluation results.

[1484] How it works: The server uses a video editing engine (e.g., FFmpeg or OpenShot) to combine audio and visual material into a single video file. It also adds transition effects and text overlays to enhance the visual quality. It also adds effects and music that suit the user's emotions.

[1485] Output: The generated video file.

[1486] Step 8:

[1487] The server provides a preview of the generated video.

[1488] Input: Generated video files, emotion evaluation results.

[1489] How it works: The server provides the user with a link to preview the video in their browser. The user reviews the video content and makes corrections as needed. The emotion engine again recognizes the user's emotions and evaluates their reactions during the preview.

[1490] Output: User's correction instructions, updated emotion evaluation results.

[1491] Step 9:

[1492] The server provides the final video.

[1493] Input: Generated video file, correction instructions from user.

[1494] How it works: The server re-edits the video based on the correction instructions and generates the final version. It then provides the final video to the user and generates a download link and a sharing link.

[1495] Output: Final video file, secure download link, and share link.

[1496] (Application example 2)

[1497] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1498] In conventional video generation systems, the process of analyzing the content of materials and creating videos is uniformly created without considering the user's emotional state, making it difficult to provide personalized content. Furthermore, in order for users to optimally understand and enjoy the content of videos, adjustments based on the user's emotions are necessary, but there has been a lack of systems that can achieve this.

[1499] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1500] In this invention, the server includes means for uploading materials, means for analyzing the content of the uploaded materials, means for generating an explanatory scenario based on the analyzed content, means for creating a narration voice and visual materials based on the generated scenario, means for generating a video by integrating the narration voice and visual materials, means for previewing the generated video and accepting correction instructions, means for recognizing the emotional state of the user and adjusting the scenario and visual style according to the emotion, and means for providing the user with the final generated video, thereby making it possible to provide optimal content according to the user's emotion.

[1501] "Means for uploading materials" refers to a device or method for uploading materials owned by a user into the system.

[1502] "Means for analyzing the content of uploaded material" means a device or method used to computerize and understand the content of uploaded material.

[1503] The "means for generating a commentary scenario" is a device or method for creating a plot or text for explanation or narration based on the analyzed content.

[1504] The "means for creating narration audio and visual materials" refers to a device or method for creating audio data and visual materials based on a generated scenario.

[1505] "Means for generating video" refers to a device or method for combining narration audio and visual materials to edit and generate a single video.

[1506] The "means for previewing the generated video and accepting correction instructions" refers to a device or method that allows a user to view the generated video in advance and indicate any necessary corrections.

[1507] "Means for recognizing the user's emotional state and adjusting the scenario and visual style according to that emotion" refers to a device or method that reads the user's emotions from their facial expressions, voice, etc., and changes or adjusts the content of the scenario and the visual design based on that.

[1508] The "means for providing the final video to the user" refers to a device or method that allows the user to download the completed video or share it with others.

[1509] An embodiment of the present invention will be described. This system analyzes materials uploaded by users and generates optimal explanatory videos while recognizing their emotional state. This system makes it possible to provide personalized video content that reflects the emotions of each individual user.

[1510] Uploading and Loading Materials

[1511] Users access the system's upload screen using a smartphone or computer browser. On the upload screen, users select and upload documents in PDF or text file format. At the same time as uploading, the system uses the device's built-in camera and microphone to recognize the user's emotional state through a facial recognition system or voice input. This emotion recognition uses facial expression recognition software and voice analysis software.

[1512] Text analysis of materials

[1513] The server receives the uploaded documents and extracts the text data using a PDF parser and OCR technology. This converts the text information from the images in the documents into text. Key information is then extracted using natural language processing (NLP). At this stage, the server also adjusts the analysis results based on the user's emotional state.

[1514] Scenario Generation

[1515] The server generates a commentary scenario for the user based on the emotional state and the analyzed text data. The generated scenario includes the flow of the commentary, the narration content, and the insertion points of visual materials. The emotion engine evaluates the user's emotions and adjusts the content of the scenario. For example, if the user is having fun, it will use casual language and select visually bright materials.

[1516] Creation of narration audio and visual materials

[1517] The server uses text-to-speech (TTS) technology to generate narration based on the generated scenario. At the same time, it creates visual materials such as illustrations and diagrams using generative AI models and image editing tools. At this stage, it also selects the style and tone that best suits the user's emotions.

[1518] Video Generation

[1519] The server combines the generated narration audio with the visual material and generates a video using a video editing engine (e.g., FFmpeg or OpenShot). The narration and visual material are placed on a timeline, and the visual quality is improved by adding transition effects and text overlays. Again, visual effects and music are applied based on the instructions of the emotion engine.

[1520] Preview and edit your video

[1521] The video generated by the server is previewed by the user. The user checks the video in their browser to ensure that the content is as intended. At the same time, the emotion engine evaluates the user's emotional response to the video. If necessary, the user can specify corrections and the video is re-edited.

[1522] Download or share the video

[1523] The final video is then delivered to the user via the server, who can either download the video or generate a shareable link to share it with others. The server generates a secure download link and also provides a viewing link for internal or external sharing.

[1524] Specific examples

[1525] For example, if a parent wants to create a learning video for their child on "Earth Geography," the system will automatically detect the parent's emotional state when they upload the materials. If a happy emotion is detected, a casual and fun-toned commentary scenario will be generated. Brightly colored diagrams and animations will be selected, and the parent can fine-tune the video while previewing it. Finally, a fun-toned geography learning video will be generated, which the parent can show to their child.

[1526] Prompt Sentence Examples

[1527] "Analyze this material and generate fun narration and colorful visuals to create a geography learning video for children."

[1528] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1529] Step 1:

[1530] Users access the system's upload screen using their devices, select and upload documents in PDF or text file format. The input data is the document file owned by the user, and the output data is the document file sent to the server. The user's emotional state is also collected at the same time using a facial recognition system and voice input. The input data is the user's facial expressions and voice, and the output data is the emotion recognition results.

[1531] Step 2:

[1532] The server analyzes the received document file. In the case of PDF, it uses a PDF parser to extract the text data and OCR technology to convert the characters in the image into text. The input data is the document file sent to the server, and the output data is the extracted text data.

[1533] Step 3:

[1534] The server uses natural language processing (NLP) techniques to extract keywords and important sentences from the extracted text data. The input data is the extracted text data, and the output data is a list of keywords and important sentences.

[1535] Step 4:

[1536] The server generates an explanatory scenario based on the emotion recognition results and the analyzed text data. The input data is a list of keywords and important sentences and the emotion recognition results, and the output data is the generated explanatory scenario. The emotion recognition results affect the tone and content of the scenario. For example, a casual scenario is generated for a positive emotional state.

[1537] Step 5:

[1538] The server uses text-to-speech (TTS) technology to create a narration voice based on the generated scenario. The input data is the generated commentary scenario, and the output data is a narration voice file. The TTS engine converts the text into speech.

[1539] Step 6:

[1540] The server creates visual materials using generative AI models and image editing tools. The input data is the generated explanatory scenario, and the output data is a set of visual materials. Diagrams and illustrations are generated to match the key points of the scenario.

[1541] Step 7:

[1542] The server uses a video editing engine (e.g., FFmpeg or OpenShot) to combine the narration audio and visual materials to generate a video. The input data is a narration audio file and a set of visual materials, and the output data is the generated video file. Transition effects and text overlays are also added.

[1543] Step 8:

[1544] The server allows the user to preview the generated video. The user checks the video on the browser and provides feedback on any corrections. The input data is the generated video file, and the output data is the user's feedback. The emotion engine reevaluates the user's emotions about the video and identifies areas that need to be re-edited.

[1545] Step 9:

[1546] The server re-edits the video based on the user's feedback and makes final adjustments according to the emotions. The input data are the user's feedback and the emotion re-evaluation results, and the output data is the final video file.

[1547] Step 10:

[1548] The server provides the final video to the user, who can then download it or share it with others via a sharing link. The input data is the final video file, and the output data is a download link and a sharing link. The server generates a secure download link and provides a viewing link.

[1549] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1550] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1551] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1552] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1553] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1554] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1555] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1556] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1557] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1558] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1559] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1560] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1561] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1562] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1563] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1564] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1565] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1566] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1567] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1568] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1569] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1570] The following is further disclosed regarding the above embodiment.

[1571] (Claim 1)

[1572] A means of uploading materials;

[1573] means for analyzing the content of the uploaded materials;

[1574] A means for generating an explanatory scenario based on the analyzed content;

[1575] a means for creating narration audio and visual materials based on the generated scenario;

[1576] A means for integrating narration audio and visual materials to generate video;

[1577] A means for previewing the generated video and receiving correction instructions;

[1578] a means for providing the final generated video to a user;

[1579] A system including:

[1580] (Claim 2)

[1581] 10. The system of claim 1, wherein natural language processing techniques are used to analyze the content of the uploaded materials.

[1582] (Claim 3)

[1583] The system according to claim 1, wherein a narration voice is created using a voice synthesis technique based on the generated scenario.

[1584] "Example 1"

[1585] (Claim 1)

[1586] A means of uploading materials;

[1587] means for analyzing the content of the uploaded materials;

[1588] A means for generating an explanatory scenario based on the analyzed content;

[1589] a means for creating narration audio and visual materials based on the generated scenario;

[1590] A means for integrating narration audio and visual materials to generate video;

[1591] A means for previewing the generated video and receiving correction instructions;

[1592] a means for providing the final generated video to a user;

[1593] A system including:

[1594] (Claim 2)

[1595] 10. The system of claim 1, wherein natural language processing techniques are used to analyze the content of the uploaded materials.

[1596] (Claim 3)

[1597] The system according to claim 1, wherein a narration voice is created using a voice synthesis technique based on the generated scenario.

[1598] (Claim 4)

[1599] 10. The system of claim 1, which uses image generation techniques to create visual materials based on the generated scenario.

[1600] (Claim 5) ...

Claims

1. A means of uploading materials; means for analyzing the content of uploaded materials; A means for generating an explanatory scenario based on the analyzed content; a means for creating narration audio and visual materials based on the generated scenario; A means for integrating narration audio and visual materials to generate video; A means for previewing the generated video and receiving correction instructions; a means for providing the final generated video to a user; A system including:

2. 10. The system of claim 1, wherein natural language processing techniques are used to analyze the content of the uploaded materials.

3. The system according to claim 1, wherein a narration voice is created based on the generated scenario using a voice synthesis technique.

Images