System

Abstract

A system is provided.SOLUTION: A system comprising: means for parsing text in an instruction manual; means for summarizing the parsed text using a generative AI model; means for converting the summarized text into video and audio formats; and means for providing the converted content to user terminals.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional instruction manuals are thick and complex, making it difficult for users to quickly and accurately grasp the information they need. For this reason, there has been a demand for a way to convert them into a simple, easy-to-understand format to aid user understanding and improve convenience. Another challenge has been to increase the reusability of the manuals by providing them visually and audibly. [Means for solving the problem]

[0005] To solve this problem, the present invention provides the following technical means. First, the system uses a means for analyzing the text of an instruction manual, followed by a means for summarizing the analyzed text using a generative AI model. The system also provides a means for converting the summarized text into video and audio formats, and finally includes a means for providing the converted content to a user terminal. Furthermore, the system includes a means for removing unnecessary information from the analyzed text and a means for converting the summarized text into an audio file using a text-to-speech engine, thereby facilitating user understanding and significantly improving convenience.

[0006] An "instruction manual" is a document that describes how to use a product, its functions, handling precautions, etc.

[0007] "Means of analysis" refers to software or algorithms that analyze text data or files and convert their contents into an understandable form.

[0008] A "generative AI model" is an artificial intelligence model trained for natural language processing and data generation, and capable of summarizing and generating text.

[0009] A "summarization technique" is a process or technique for extracting the essential parts of an original text and summarizing them in a concise form.

[0010] "Means for converting into video format" refers to technologies and tools that generate images and videos in order to provide text data or summaries as visual content.

[0011] "Means for converting to audio format" refers to technology or software that converts text data into audio data and provides information audibly.

[0012] The "means for providing to the user terminal" refers to a technology or system that transmits the generated content to the user's device so that the user can view or listen to it.

[0013] A "clutter remover" is a process or algorithm that removes unnecessary parts from the analyzed text, leaving only the important information.

[0014] A "text-to-speech engine" is software that synthesizes input text data into natural-sounding speech and provides information in audio form. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention relates to a system that uses a generative AI model to summarize instruction manuals and convert them into video and audio formats. The purpose of this system is to provide users with instruction manuals in a format that is easy to understand. Specific embodiments of this system are described below.

[0037] User Operation

[0038] Users access the system to upload instruction manual files (PDF or text files). Users click the "Upload" button in their web browser or mobile application and select the instruction manual they want to upload from the file selection dialog.

[0039] Terminal operation

[0040] The device reads the instruction manual file selected by the user and sends the binary data to the server. Specifically, it sends the file to the server as an HTTP POST request, requesting that the server analyze the instruction manual.

[0041] Server Processing

[0042] The server receives the file sent from the terminal and analyzes its contents. For analysis, an appropriate library (e.g., a PDF analysis library or text analysis algorithm) is used. For PDF files, text extraction is performed, and for text files, the text data is read directly. Next, the plain text is preprocessed to remove unnecessary information and special characters.

[0043] Next, we use a generative AI model to summarize the parsed text. We use a generative AI model (e.g., GPT-4) to extract key points from the instruction manual and generate a concise summary. We evaluate the summary results and regenerate them if necessary.

[0044] Content Conversion and Delivery

[0045] The summarized text is then converted into video and audio formats. The server first inputs the summarized text into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. Then, video generation software is used to generate an animated video, which includes slides and visuals based on the summarized content. The generated audio file is also incorporated into the video.

[0046] Finally, the generated video and audio files are uploaded to cloud storage, and a download link or streaming URL is provided to the user's device, allowing the user to view the video and audio and understand the contents of the instruction manual.

[0047] Specific examples

[0048] For example, consider the case where a user wants to upload an instruction manual for a home appliance (e.g., a washing machine) to the system. The user accesses a web application and clicks the "Upload" button. The user selects a PDF file of the washing machine's instruction manual from the local storage and uploads it. The device then sends this PDF file to the server, which extracts text from the PDF and performs preprocessing.

[0049] The server uses a generative AI model to extract only the key information and generate a summary divided into sections such as "Initial Setup," "How to Use," and "Troubleshooting." Based on this summary, a text-to-speech engine is used to generate an audio file, and video generation software is used to create an animated video.

[0050] Finally, the server uploads the generated video and audio files to cloud storage and provides a download link to the user. The user can then use the provided link to watch the video and understand how to set up and use the washing machine. In this way, a system is realized that converts complex instruction manuals into a user-friendly format, facilitating user understanding.

[0051] The processing flow will be explained below.

[0052] Step 1:

[0053] The user starts uploading the instruction manual.

[0054] Users click the "Upload" button in their web browser or mobile application and select the instruction manual (PDF or text file) they want to upload from the file selection dialog.

[0055] Step 2:

[0056] The device sends the uploaded file to the server.

[0057] The terminal reads the selected file, converts it to binary data, and then generates an HTTP POST request and sends it to the server with the file data.

[0058] Step 3:

[0059] The server receives and parses the file.

[0060] The server receives the HTTP request and extracts the attached file data. For PDF files, the server extracts the text using a PDF parsing library. For text files, the server reads the text data directly. The extracted text data is then preprocessed to remove unnecessary whitespace and special characters.

[0061] Step 4:

[0062] The server generates the summary using the generative AI model.

[0063] The server loads the generative AI model and feeds it preprocessed text data. The AI ​​model extracts key information from this data and generates a concise summary. The summary is evaluated and regenerated if necessary.

[0064] Step 5:

[0065] The server converts the summary into video or audio format.

[0066] The server inputs the summarized text into a text-to-speech engine to generate an audio file, then uses video generation software to generate an animated video based on the summarized content, adding text slides and visuals to the video and incorporating the generated audio file.

[0067] Step 6:

[0068] The server provides the generated video and audio files to the user.

[0069] The server uploads the generated video and audio files to cloud storage and generates and sends a download link or streaming URL to the user's device.

[0070] Step 7:

[0071] View and listen to user-generated content.

[0072] Users can download or stream the video and audio using the provided link, and can pause, rewind, or fast-forward to the section they want while watching.

[0073] Through these steps, a system is realized that converts the contents of an instruction manual into a concise, easy-to-understand video or audio format and provides it to users.

[0074] Example 1

[0075] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0076] The instruction manuals that accompany a wide variety of modern products are often complex and require time and effort for users to understand. In particular, if they contain long texts or a lot of technical terminology, it can be difficult for users to understand, and they may not use or maintain the product properly. Therefore, there is a need for a method to communicate the contents of the instruction manual efficiently and effectively.

[0077] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0078] In this invention, the server includes a means for analyzing data of the instruction manual, a means for summarizing the analyzed data using a generative AI model, and a means for converting the summarized data into audio and video formats, thereby enabling the user to understand the contents of the instruction manual concisely and efficiently.

[0079] An "instruction manual" is a document that provides information on how to use a product or service, installation procedures, maintenance methods, troubleshooting, etc.

[0080] "Data" means content expressed in the form of information, instructions, numbers, etc., that is stored and transmitted electronically.

[0081] "Analysis" is the process of breaking down complex data and information and converting it into an understandable form.

[0082] A "generative AI model" is an artificial intelligence algorithm that uses deep learning and natural language processing to generate new data and text.

[0083] A "summary" is a concise summary of a large amount of information, extracting only the main points and shortening them.

[0084] "Speech" is the electronic representation of spoken words or other acoustic signals.

[0085] "Video" refers to the electronic representation of visual information, including text, images, animations, etc.

[0086] "User" means a person or entity that uses a system or product.

[0087] A "terminal" is an electronic device that allows a user to access and operate the system.

[0088] "Offering" is the act of giving something to someone else or making it available for their use.

[0089] The present invention relates to a system that uses a generative AI model to summarize instruction manuals and convert them into audio and video formats. The purpose of this system is to provide users with instruction manuals in a format that is easy to understand. Specific embodiments of this system are described below.

[0090] User Operation

[0091] Users access the system to upload instruction manual files (PDF or text files). They click the "Upload" button in their web browser or mobile application and select the instruction manual they want to upload from the file selection dialog. Once the upload is complete, the device prepares to send the file to the server.

[0092] Terminal operation

[0093] The device loads the instruction manual file selected by the user and sends the binary data to the server. Specifically, the device creates an HTTP POST request to send the selected file to the server as binary data. This request also includes metadata such as the file name and file type.

[0094] Server Processing

[0095] The server receives the file sent from the terminal. After receiving it, it identifies the file type and selects the appropriate parsing method. If it is a PDF file, it uses a PDF parsing library (e.g., PDFBox or PyPDF2) to extract the text data. If it is a text file, it reads the text data directly.

[0096] The server then preprocesses the extracted text data, removing unnecessary information (e.g., advertisements or specific product brands) and special characters, and checking the accuracy of grammar and syntax, correcting errors if necessary.

[0097] The server inputs the preprocessed text into a generative AI model (e.g., OpenAI's GPT-4) to generate a summary. Specifically, it invokes the generative AI model and gives it the following prompt:

[0098] Prompt: "Summarize the text from the instruction manual below. Extract the key points and summarize them concisely."

[0099] The generative AI model extracts important information based on the input text and generates a summary.

[0100] Content Conversion and Delivery

[0101] The server then converts the generated summary into audio and video formats. First, the summarized text is fed into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. Then, video generation software (e.g., Adobe After Effects or Apple Motion) is used to create an animated video based on the audio data. Slides and visuals based on the summarized content are added to the video.

[0102] Finally, the server uploads the generated video and audio files to cloud storage (e.g., Amazon S3 or Google Cloud Storage), obtains the file URL on the cloud storage, and provides a download link or streaming URL to the user's device.

[0103] Users can use the links to view videos and audio, which allows them to easily understand the contents of the instruction manual through both sight and sound.

[0104] Specific examples

[0105] For example, consider the case where a user wants to upload an instruction manual for a home appliance (e.g., a washing machine). The user accesses a web application, clicks the "Upload" button, selects the PDF file of the washing machine's instruction manual from the local storage, and uploads it. The device then sends this PDF file to the server, which extracts text from the PDF and performs preprocessing.

[0106] The server uses a generative AI model to extract only the key information and generate a summary divided into sections such as "Initial Setup," "How to Use," and "Troubleshooting." Based on this summary, a text-to-speech engine is used to generate an audio file, and video generation software is used to create an animated video.

[0107] Finally, the server uploads the generated video and audio files to cloud storage and provides a download link to the user. The user can then use the provided link to watch the video and understand how to set up and use the washing machine. This system makes it possible to provide complex instruction manuals in a concise, visually easy-to-understand format.

[0108] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0109] Step 1: User interaction

[0110] Users access the system and upload instruction manual files (PDF or text files). Specifically, they click the "Upload" button in their web browser or mobile application and select the instruction manual they want to upload from the file selection dialog. The input is the instruction manual file specified by the user, and the output is the binary data of the selected file.

[0111] Step 2: Loading and sending the file

[0112] The terminal reads the instruction manual file selected by the user and sends the binary data to the server. Specifically, the terminal creates an HTTP POST request and sends the selected file as binary data to the server. The input is the binary data of the file selected by the user, and the output is the HTTP POST request sent to the server.

[0113] Step 3: Receiving and analyzing the file

[0114] The server receives the file sent from the terminal. After receiving it, it identifies the file type and selects the appropriate parsing method. If it is a PDF file, it uses a PDF parsing library (e.g., PDFBox or PyPDF2) to extract the text data. If it is a text file, it reads the text data directly. The input is the file data received by the server, and the output is the extracted text data.

[0115] Step 4: Preprocessing the data

[0116] The server preprocesses the extracted text data. This preprocessing removes unnecessary information (e.g., advertisements or specific product brands) and special characters. It also checks the accuracy of grammar and syntax and corrects errors as needed. The input is the extracted text data, and the output is the preprocessed, clean text data.

[0117] Step 5: Summarizing the text

[0118] The server inputs the preprocessed text into a generative AI model (e.g., GPT-4) to generate a summary. Specifically, it invokes the generative AI model and gives it the following prompt:

[0119] Prompt: "Summarize the text from the instruction manual below. Extract the key points and summarize them concisely."

[0120] The input is preprocessed text data and a prompt sentence, and the output is the generated summary sentence.

[0121] Step 6: Converting the summary into audio and video formats

[0122] The server inputs the generated summary into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. It then uses video generation software (e.g., Adobe After Effects or Apple Motion) to create an animated video, adding slides and visuals based on the summarized content. The input is the summary, and the output is the generated audio and video files.

[0123] Step 7: Upload and provide files

[0124] The server uploads the generated video and audio files to cloud storage (e.g., Amazon S3 or Google Cloud Storage). It obtains the file URL on the cloud storage and provides a download link or streaming URL to the user's device. The input is the generated audio and video files, and the output is the download link or streaming URL provided to the user.

[0125] Specific examples

[0126] For example, if a user wants to upload a washing machine instruction manual, the actual operation steps are as follows:

[0127] Step 1: The user accesses the web application, clicks the "Upload" button, selects the washing machine instruction manual (PDF file), and clicks the "Submit" button.

[0128] Step 2: The device reads the selected PDF file and creates an HTTP POST request to send to the server.

[0129] Step 3: The server receives the file and uses PDFBox to extract and preprocess the text data.

[0130] Step 4: The server inputs the preprocessed text into the generative AI model to generate a summary.

[0131] Step 5: The server feeds the summary into Google Text-to-Speech to generate an audio file, and then uses Adobe After Effects to generate a video of the slides and audio synchronized to the text.

[0132] Step 6: The server uploads the generated video and audio files to Amazon S3 and provides a download link to the user.

[0133] Step 7: The user uses the provided link to watch and listen to videos and audio to understand how to set up and use the washing machine.

[0134] This series of processes provides the instruction manual in a concise and visually easy-to-understand format.

[0135] (Application example 1)

[0136] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0137] Conventional instruction manuals require a huge amount of time and effort for users to understand their contents, which is a major obstacle to efficient work, especially on factory floors. Furthermore, paper-based or simple text-based manuals make it difficult to visually understand procedures and operations, increasing the risk of operational errors and maintenance mistakes. This can lead to reduced production efficiency and safety, so a more user-friendly instruction manual provision system is needed.

[0138] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0139] In this invention, the server includes a means for analyzing the text of the instruction manual, a means for summarizing the analyzed text using a generative AI model, a means for converting the summarized text into video and audio formats, a means for providing the summarized content to a user via smart glasses in visual and audio formats, and a means for providing the converted content to a user terminal, thereby enabling the user to understand the instruction manual and maintenance procedures of a factory robot in real time through visual and audio formats, thereby achieving efficient and safe work.

[0140] The "means for analyzing the text of the instruction manual" has the function of reading the document data of the instruction manual uploaded by the user and analyzing its contents.

[0141] "Means for summarizing analyzed text using a generative AI model" refers to a function that has the function of concisely summarizing the contents of analyzed text data through a generative AI model that uses natural language processing technology.

[0142] A "means for converting summarized text into video and audio formats" is a means for using a text-to-speech engine or video generation software to create audio files and animated videos based on the summarized text.

[0143] The "means for visually and audibly providing summarized content to a user via smart glasses" means a means for providing summarized text content to the user's visual and auditory senses in real time via smart glasses.

[0144] The "means for providing converted content to a user terminal" refers to a function that uploads the generated video and audio content to cloud storage and provides a download link or streaming URL to the user terminal.

[0145] The "means for removing unnecessary information" has a function of performing preprocessing to remove unimportant information and special characters that become noise from the analyzed text data.

[0146] "Means for converting to an audio file using a text-to-speech engine" means a means for converting the summarized text into audio using text-to-speech technology and generating an audio file.

[0147] The "means of converting the text into an animated video using video generation software" has the function of generating an animated video based on the summarized text, along with slides and visuals.

[0148] The "means for providing the contents of the factory robot's instruction manual in real time" has the function of presenting information on the handling and maintenance of the factory robot to workers in real time through smart glasses.

[0149] MODE FOR CARRYING OUT THE INVENTION

[0150] The present invention relates to a system for providing instruction manuals in a format that is easy for users to understand. This system analyzes the text of the instruction manual, converts the summarized content into video and audio formats, and provides the content in real time via smart glasses. An embodiment of this system will be specifically described below.

[0151] Program Overview

[0152] The system has the following main functions:

[0153] 1. Ability to analyze instruction manual text

[0154] 2. Ability to summarize text using generative AI models

[0155] 3. Ability to convert summarized text into video and audio formats

[0156] 4. Function to provide content to smart glasses and user devices

[0157] Hardware and Software Configuration

[0158] The hardware and software used includes:

[0159] Terminal: The device to which the user uploads the instruction manual, such as a PC or smartphone.

[0160] Server: A central server that performs text analysis, summarization, and content conversion. It uses Python, a PDF analysis library, GPT-4, a Text-to-Speech API, and a video generation API as its main software.

[0161] Smart glasses: devices that provide real-time visual and audio information, such as Google Glass.

[0162] Data processing and calculation

[0163] 1. Upload and analyze the instruction manual:

[0164] Users upload a PDF of the instruction manual through their device, and the server extracts the text from the PDF and removes unnecessary information.

[0165] 2. Summary generation:

[0166] The extracted text is summarized using a generative AI model (GPT-4). As a concrete example, the following prompt sentence is sent to the AI ​​model to generate a summary:

[0167] Please summarize the contents of the following instruction manual:

[0168] "This robot requires daily inspection. The inspection procedure is as follows: remove part A, clean part B, and reinstall part C."

[0169] 3. Content Transformation:

[0170] The summarized text is converted into audio using a Text-to-Speech API, and then an animated video is generated using a Video Generation API.

[0171] 4. Provision to smart glasses and user devices:

[0172] The generated audio and video content is uploaded to cloud storage and provided to users in real time via smart glasses, and a download link or streaming URL is provided to the user's device as needed.

[0173] Specific examples

[0174] For example, when a factory worker wearing smart glasses performs maintenance on a robot, he or she uploads a PDF instruction manual to the system. The server extracts text from the PDF and generates a summary using GPT-4. The summary is then converted into a video with audio and animation using the Text-to-Speech API and the Video Generation API. This audio and video is then provided to the worker in real time via the smart glasses. This allows the worker to visually confirm the procedure while receiving audio instructions.

[0175] This system will enable factory workers to significantly reduce the time it takes to understand complex instruction manuals, enabling them to work efficiently and safely.

[0176] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0177] Step 1:

[0178] A user accesses the web or mobile application. The user clicks the "Upload" button, selects the instruction manual file (PDF or text file), and uploads it.

[0179] Input: Instruction manual file

[0180] Output: Binary data of the instruction manual

[0181] Specific operation: The user selects the desired instruction manual from the file selection dialog and presses the upload button. This action sends the binary data of the instruction manual to the terminal.

[0182] Step 2:

[0183] The terminal transmits the binary data of the instruction manual selected by the user to the server.

[0184] Input: Binary data of instruction manual

[0185] Output: Data sent to the server

[0186] Specific operation: The device sends the binary data of the instruction manual to the server as an HTTP POST request. This request uploads the file to the server.

[0187] Step 3:

[0188] The server analyzes the received instruction manual file and extracts the text data. If it is a PDF file, it uses a PDF analysis library to extract the text, and if it is a text file, it reads it directly.

[0189] Input: Binary data of the instruction manual uploaded to the server

[0190] Output: Extracted text data

[0191] What it does: The server uses a PDF parsing library to extract text from PDF files, or if it's a text file, reads the text data directly.

[0192] Step 4:

[0193] The server preprocesses the extracted text data to remove unnecessary information and special characters.

[0194] Input: Extracted text data

[0195] Output: Preprocessed clean text data

[0196] Specific operation: The server cleanses the text data using regular expressions and string manipulation to remove unnecessary information and special characters.

[0197] Step 5:

[0198] The server inputs the preprocessed text data into a generative AI model (GPT-4) to generate a summary.

[0199] Input: Preprocessed clean text data

[0200] Output: Summary text

[0201] Specific operation: The server inputs the following prompt to the generated AI model:

[0202] Please summarize the contents of the following instruction manual:

[0203] "This is an instruction manual divided into sections for setup, usage, and troubleshooting. The setup is as follows..."

[0204] A generative AI model generates a summary based on this prompt.

[0205] Step 6:

[0206] The server converts the generated summary text into audio using a Text-to-Speech API.

[0207] Input: Summary text

[0208] Output: Audio file (mp3, etc.)

[0209] Specific operation: The server sends the summary text to the Text-to-Speech API and receives the generated audio file.

[0210] Step 7:

[0211] The server generates animated videos based on the summary text using a video generation API.

[0212] Input: Summary text

[0213] Output: Animated video file

[0214] Specific operation: The server sends the summary text to the video generation API and receives the generated video file.

[0215] Step 8:

[0216] The server uploads the generated audio and video files to cloud storage and generates a link to provide to the user terminal and smart glasses.

[0217] Input: Audio and video files

[0218] Output: Upload and download links to cloud storage, streaming URL

[0219] Specific operation: The server uploads the audio and video files to a cloud storage service and provides the generated link to the user's application and smart glasses app.

[0220] Step 9:

[0221] Users can view and listen to the contents of the instruction manual summarized in real time through smart glasses and devices.

[0222] Input: Download link / Streaming URL

[0223] Output: Real-time viewing and listening content

[0224] What it does: The user clicks on the provided link or accesses the viewing screen with smart glasses to view audio and video content in real time.

[0225] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0226] The present invention is a system that not only summarizes instruction manuals using a generative AI model and converts the summaries into video and audio formats, but also combines it with an emotion engine that recognizes user emotions to provide more user-friendly content. Specific embodiments of this system are described below.

[0227] User Operation

[0228] First, users access the system and upload the instruction manual file (PDF or text file). Users click the "Upload" button in their web browser or mobile application and select the instruction manual they want to upload from the file selection dialog.

[0229] Terminal operation

[0230] The device reads the instruction manual file selected by the user, converts it into binary data, and then sends the file data to the server as an HTTP POST request to request analysis.

[0231] Server Processing

[0232] The server receives the file sent from the device and parses its contents. This involves extracting text from the PDF file using a PDF parsing library, or reading the text directly if it is a text file. The server then preprocesses the plain text, removing unnecessary whitespace and special characters.

[0233] The parsed text is then summarized using a generative AI model (e.g., GPT-4) to extract key points from the instruction manual and generate a concise summary. The final summary is evaluated and regenerated if necessary.

[0234] Content Conversion and Delivery

[0235] The summarized text is then converted into video or audio format. The server first inputs the summarized text into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. Then, video generation software is used to generate an animated video based on the summarized content. This video includes text slides and visuals, and also integrates the generated audio file.

[0236] Incorporating an emotion engine

[0237] The emotion engine recognizes the user's emotional state in real time and adjusts the content of the summary text and the tone and speed of the generated video and audio based on its output. For example, if the user feels confused or anxious about the information, the server will provide more detailed summary content at a slower pace based on the output of the emotion engine. The data from the emotion engine will also be used to adjust the animation and audio tone to reduce the user's psychological burden.

[0238] Content Submissions and Feedback

[0239] Finally, the server uploads the generated video and audio files to cloud storage and generates and sends a download link or streaming URL to the user's device. The user uses this link to watch the video or audio. The emotion engine monitors the user's reactions in real time while watching and adjusts the content as needed.

[0240] Specific examples

[0241] For example, consider a case where a user uploads a user manual for a home appliance (e.g., a smartphone). The user accesses a web application, clicks the "Upload" button, and selects the PDF file of the smartphone's user manual. The device then sends this file to a server, which extracts text from the PDF and uses a generative AI model to generate a summary of key sections.

[0242] The server then inputs the summarized text into a text-to-speech engine to generate an audio file, and uses video generation software to create an animated video. During this process, an emotion engine recognizes the user's emotions and adjusts the summary and video tone and speed as needed. For example, if the user is confused, the explanation will be more detailed and the video will be slower.

[0243] Finally, the server uploads the generated video and audio files to cloud storage and provides a download link to the user. The user clicks on this link to watch the video and understand how to set up the smartphone and use its functions. In this way, the present invention is a system that converts complex instruction manuals into a user-friendly format and provides them adaptively using an emotion engine.

[0244] The processing flow will be explained below.

[0245] Step 1:

[0246] The user starts uploading the instruction manual.

[0247] Users click the "Upload" button in their web browser or mobile application and select the instruction manual (PDF or text file) they want to upload from the file selection dialog.

[0248] Step 2:

[0249] The device sends the uploaded file to the server.

[0250] The terminal reads the selected file, converts it to binary data, then generates an HTTP POST request and sends it to the server with the file data.

[0251] Step 3:

[0252] The server receives and parses the file.

[0253] The server receives the HTTP request and extracts the attachment data. For PDF files, the server extracts the text using a PDF parsing library. For text files, the server reads the text data directly. The extracted text data is then preprocessed to remove unnecessary whitespace and special characters.

[0254] Step 4:

[0255] The server generates the summary using the generative AI model.

[0256] The server loads the generative AI model and feeds it preprocessed text data. The AI ​​model extracts key information from this data and generates a concise summary. The generated summary is evaluated and regenerated if necessary.

[0257] Step 5:

[0258] The server uses an emotion engine to recognize the user's emotional state.

[0259] The server uses an emotion engine to collect and analyze the user's real-time emotion data, such as facial expressions, tone of voice, and eye movements, to understand the user's emotional state.

[0260] Step 6:

[0261] The server converts the summary into video or audio format.

[0262] The server inputs the summarized text into a text-to-speech engine to generate an audio file, and then uses video generation software to generate an animated video based on the summarized content, adjusting the tone and speed of the video and audio accordingly based on data from the emotion engine.

[0263] Step 7:

[0264] The server provides the generated video and audio files to the user.

[0265] The server uploads the generated video and audio files to cloud storage and generates and sends a download link or streaming URL to the user's device.

[0266] Step 8:

[0267] View and listen to user-generated content.

[0268] Users can download or stream the video and audio using the provided link, and can pause, rewind, or fast-forward to the section they want while watching.

[0269] Step 9:

[0270] The server continuously monitors the user's emotional state.

[0271] The server monitors the user's emotional state in real time while watching video or listening to audio, collecting feedback and adjusting or regenerating content as needed based on this data.

[0272] This makes it possible to realize a system that provides the contents of the instruction manual to the user in an optimal format and makes it easy to understand.

[0273] Example 2

[0274] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0275] Conventional instruction manuals are often complex and difficult to understand, making it difficult for users to quickly obtain the information they need. Furthermore, conventional technologies provide information in a uniform format without considering the user's emotional state, making it difficult to provide a user-friendly experience. Furthermore, technologies for converting instruction manual summaries into video or audio formats are limited, making it difficult to meet the diverse needs of users.

[0276] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for analyzing the instruction manual document, means for summarizing the analyzed document using a generative AI model, means for converting the summarized document into video and audio format, means for uploading the converted content to cloud storage to be provided to the user terminal, and means for recognizing the user's emotional state using an emotion engine and adjusting the summary content and the tone and speed of the content. This not only enables the user to quickly obtain the information they need, but also provides the information in an appropriate format and at an appropriate pace depending on the user's emotional state, resulting in a more user-friendly experience.

[0277] An "instruction manual" is a document provided to explain how to use or operate a product or service.

[0278] A "document" is a collection of text or information written to convey a specific message.

[0279] "Analysis" is the process of examining data or information in detail to understand its structure and meaning.

[0280] A "generative AI model" is a model that uses artificial intelligence technology to generate new text or data.

[0281] A "summary" is a concise summary of the important parts of a document or data.

[0282] A "video format" is a data format that expresses movement by playing back a series of images or videos over time.

[0283] An "audio format" is a format in which sound or voice is represented as digital data.

[0284] "Cloud storage" is a remote data storage service that allows you to store and access data over the Internet.

[0285] An "emotion engine" is a technology / system that analyzes a user's emotional state in real time and utilizes the results.

[0286] A "user terminal" is hardware or a device that a user directly operates to input or receive information.

[0287] "Tone" refers to the tone, timbre, and atmosphere of a voice or text.

[0288] "Speed" refers to the speed at which information or content is provided to users.

[0289] The present invention is a system for analyzing an instruction manual, summarizing the contents thereof, and providing the summarization to a user. A specific embodiment of this system will be described below.

[0290] System Overview

[0291] Hardware Configuration

[0292] This system consists of three main components: user terminals, servers, and cloud storage. User terminals are devices such as PCs, smartphones, and tablets, while servers are computing devices for analysis and generation processing. Cloud storage is a remote data storage service that stores generated content and provides it to users.

[0293] Software Configuration

[0294] The system includes a PDF parsing library, a generative AI model (e.g., GPT-4), a text-to-speech engine (e.g., Google Text-to-Speech), video generation software, and an emotion engine.

[0295] Processing flow

[0296] User Operation

[0297] Users use a web browser or mobile application to upload instruction manual files (PDF or text files) to the system. Specifically, they access the system, click the "Upload" button, and select the instruction manual they want to upload from the file selection dialog.

[0298] Terminal operation

[0299] The device reads the instruction manual file selected by the user, converts it into binary data, and then sends the file data to the server as an HTTP POST request.

[0300] Server Processing

[0301] The server receives the file sent from the device and analyzes its contents. For PDF files, it extracts the text using a PDF analysis library, while for text files, it reads the text data directly. It then summarizes the analyzed text using a generative AI model. The generative AI model used (e.g., GPT-4) is given a prompt such as "Please briefly summarize the key points of this instruction manual" to generate an appropriate summary.

[0302] Content generation

[0303] The server inputs the summarized text into a text-to-speech engine to generate an audio file, and then uses video generation software to generate an animated video based on the summarized content, incorporating text slides and visuals and the generated audio file.

[0304] Emotion Engine Operation

[0305] The emotion engine recognizes the user's emotional state in real time. For example, if the user is using a webcam, it uses facial expression recognition technology to analyze their emotions and sends the results to the server. Based on the results, the server adjusts the content of the summary text and the tone and speed of the generated content.

[0306] Content provision

[0307] The server uploads the generated video and audio files to cloud storage and provides a download link or streaming URL to the user's device. The user uses this link to watch the video or audio. During viewing, the emotion engine monitors the user's reactions in real time and adjusts the content as needed.

[0308] Specific examples

[0309] For example, consider a user uploading a user manual for a home appliance (e.g., a smartphone). The user accesses a web application, clicks the "Upload" button, and selects the PDF file of the smartphone's user manual. The device sends this file to a server, which extracts text from the PDF and uses a generative AI model to generate a summary of key sections. The server then inputs the summarized text into a text-to-speech engine to generate an audio file. It then uses video generation software to create an animated video. During this process, an emotion engine recognizes the user's emotions and adjusts the summary content, tone, and speed of the video as needed. Finally, the server uploads the generated video and audio files to cloud storage and provides the user with a download link. The user clicks the link to watch the video and learn how to set up the smartphone and use its features.

[0310] Prompt Sentence Examples

[0311] "Please briefly summarize the important points of this instruction manual."

[0312] The above is a detailed description of the embodiment of the present invention. This system can convert complex instruction manuals into a user-friendly format and provide them adaptively using an emotion engine.

[0313] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0314] Step 1:

[0315] The user uploads an instruction manual to the system. The user accesses the system, clicks the "Upload" button using a web browser or mobile application, and selects the instruction manual file (PDF or text file) from the file selection dialog. The input is the instruction manual file, and the output is the path to the selected file.

[0316] Step 2:

[0317] The terminal reads the selected file and converts it to binary data. The terminal uses the file system API to read the specified file and converts it to binary data. It then sends this binary data to the server as an HTTP POST request. The input is the path of the file selected by the user, and the output is the binary data sent to the server.

[0318] Step 3:

[0319] The server receives the binary data sent from the terminal, saves the file, and analyzes the contents. The server first saves the file in a temporary directory. Next, if it is a PDF file, it uses a PDF parsing library to extract the text, or if it is a text file, it reads the text data directly. The input is binary data, and the output is the extracted plain text. Specific operations include calling the API of the PDF parsing library to extract the text from the file.

[0320] Step 4:

[0321] The server summarizes the extracted text using a generative AI model. The server preprocesses the extracted plain text to remove unnecessary white space and special characters. The generative AI model is then given a prompt sentence, "Please briefly summarize the important points of this instruction manual," to generate summarized text. The input is the parsed plain text, and the output is the summarized text.

[0322] Step 5:

[0323] The server converts the summarized text into audio format. The server inputs the summarized text into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. The input is the summarized text, and the output is an audio file. Specifically, the Google Text-to-Speech API is used to convert text into audio.

[0324] Step 6:

[0325] The server generates a video that includes an audio file. The server uses video generation software to create an animated video by integrating text slides and visuals with the generated audio file. The input is the summarized text and audio file, and the output is the generated video file. Specifically, the video generation software runs a script to generate the video.

[0326] Step 7:

[0327] The emotion engine recognizes the user's emotional state in real time. If the user is using a webcam, the emotion engine uses facial expression recognition technology to analyze emotions and sends the results to the server. The input is the video feed from the webcam, and the output is the analyzed emotion data. Specifically, it runs a facial expression recognition algorithm to analyze the user's emotions.

[0328] Step 8:

[0329] The server adjusts the content based on the output of the emotion engine. The server uses the emotion data to adjust the content of the summary text and the tone and speed of the generated video and audio. The input is emotion data, and the output is the adjusted content. Specifically, the server changes the parameters of the generation algorithm in real time.

[0330] Step 9:

[0331] The server uploads the generated video and audio files to cloud storage. The server uploads the files using a cloud storage service and provides an access link to the user's device. The input is the generated content file, and the output is the upload result to cloud storage and the access link. Specifically, the server uses the cloud storage API to upload the files and generate a link.

[0332] Step 10:

[0333] The user views the content using the provided link. The user clicks on a download link or streaming URL to view the video or audio. The input is the access link, and the output is the content to be viewed. The specific operation is to play the content using a browser or media player.

[0334] The above is a detailed flow of the program processing for this system. The input and output at each step are clearly stated, and the type of data processing and calculation that is performed is specifically explained.

[0335] (Application example 2)

[0336] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0337] In recent years, many factories have introduced new robots, but complicated instruction manuals hinder workers' quick understanding, resulting in a decline in work efficiency. Furthermore, there is a lack of technology to adapt instructions according to the user's emotional state, which often leaves workers confused.

[0338] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing the text of the instruction manual, means for summarizing the analyzed text using a generative AI model, means for converting the summarized text into video and audio format, means for providing the converted content to a user terminal, means for recognizing the user's emotional state in real time, and means for adjusting the content of the summarized text and the tone and speed of the video and audio based on the user's emotional state. This makes it possible to improve work efficiency and reduce user confusion.

[0339] An "instruction manual" is a document that contains information on how to use, maintain, and handle a product.

[0340] "Analyzing" means breaking down given data and examining each element in detail.

[0341] A "generative AI model" is a model that uses artificial intelligence techniques to generate new data, such as summarizing or automatically generating text.

[0342] To "summarize" means to summarize a longer piece of text or information in a concise form.

[0343] "Video format" is a format that expresses information as moving images.

[0344] "Audio format" is a format in which information is conveyed auditorily.

[0345] A "user terminal" is a device such as a computer or smartphone that is directly operated by a user.

[0346] "Emotional state" refers to a user's current psychological state or mood.

[0347] "Real-time" is a term that refers to processing or response occurring immediately, as in actual time.

[0348] "Tone" is an expression that refers to the tone or color of a sound or voice.

[0349] "Speed" refers to the speed at which information is transmitted within a given period of time.

[0350] This system summarizes the complex content of instruction manuals, converts them into video and audio formats, and adaptively adjusts the content according to the user's emotions, allowing factory workers to easily understand the instruction manuals for newly installed robots and improving work efficiency.

[0351] System configuration

[0352] 1. User Operation

[0353] First, the user uploads the instruction manual (PDF or text file) using the smartphone app. Click the "Upload" button in the app and select the file to upload from the file selection dialog.

[0354] 2. Terminal operation

[0355] The uploaded file is read by the smartphone and converted into binary data, which is then sent to the server as an HTTP POST request.

[0356] 3. Server Processing

[0357] The server uses a PDF parsing library (e.g., fitz) to extract text from the PDF file, or reads it directly in the case of text files.

[0358] Then, a generative AI model (e.g., GPT-4) is used to extract key points from the plain text and generate a concise summary. A text-to-speech engine (e.g., gTTS) is then used to generate an audio file, which is then used to create an animated video using video generation software (e.g., moviepy).

[0359] 4. Emotion engine integration

[0360] The emotion engine recognizes the user's emotional state in real time through the user's camera feed. For example, if the user is confused, the server will adjust the summary, tone and speed of the video to provide a more detailed and slower explanation.

[0361] 5. Content Submissions and Feedback

[0362] The generated video and audio files are uploaded to cloud storage, and a download link or streaming URL is generated for the user. The user can use the link to watch the video or audio. The emotion engine monitors the user's reactions in real time while watching and adjusts the content as needed.

[0363] Specific examples

[0364] For example, a factory worker uploads a PDF instruction manual for a new robot to the app. The server extracts the text from the PDF and uses a generative AI model to summarize the key points of the manual. The summarized text is then fed into a text-to-speech engine to generate an audio file, which is then used by video generation software to create an animated video.

[0365] The emotion engine monitors the user's emotional state in real time and regenerates the summary content if they are confused, providing more detailed and slower explanations.

[0366] Prompt Sentence Examples

[0367] Summarize the following text:

[0368] [Text extracted from PDF]

[0369] It should be around 100-200 characters in length, focus on the key points, and be concise.

[0370] As described above, this system contributes to improving work efficiency by converting complex instruction manuals into an easily understandable format and providing adaptive explanations using an emotion engine.

[0371] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0372] Step 1:

[0373] The user uploads an instruction manual (PDF or text file) using a smartphone app. The input data is the instruction manual file on the user's device, and the output data is binary data sent to the server.

[0374] Step 2:

[0375] The terminal reads the uploaded file, converts it to binary data, and sends the converted binary data to the server as an HTTP POST request. The input is the file selected by the user, and the output is the binary data sent to the server.

[0376] Step 3:

[0377] The server parses the received binary data and extracts the text from the PDF file using a PDF parsing library (e.g., fitz) or reads the data directly from the text file. The input is the binary data sent to the server, and the output is the extracted text data.

[0378] Step 4:

[0379] The server generates a summary from the plain text using a generative AI model (e.g., GPT-4). In this process, the input is the extracted text data, and the output is the generated summary text. Specifically, a prompt sentence is input to the generative AI model, and the model performs a process of summarizing the key points.

[0380] Step 5:

[0381] The server inputs the generated summary text into a text-to-speech engine (e.g., gTTS) to generate an audio file. The input is the summarized text data, and the output is an audio file.

[0382] Step 6:

[0383] The server creates an animated video using video generation software (e.g., moviepy) based on the generated audio file. The input is the audio file and summary text, and the output is the generated animated video file.

[0384] Step 7:

[0385] The emotion engine recognizes the user's emotional state in real time through the user's camera feed. The input is data from the user's camera feed, and the output is data on the user's emotional state. Based on the emotional state, it adjusts the tone and speed of the video and audio.

[0386] Step 8:

[0387] The server realigns the summary text and the generated video and audio files according to the output data of the emotion engine. The input is the emotion engine data, and the output is the realigned content.

[0388] Step 9:

[0389] Finally, the generated video and audio files are uploaded to cloud storage, and a download link or streaming URL is provided to the user's device. The input is the generated video and audio files, and the output is uploading them to cloud storage and providing a link to the user.

[0390] A concrete example would be a factory floor worker uploading a new robot instruction manual to an app, watching abridged video and audio, and receiving detailed, slow-fire explanations when the emotion engine detects confusion.

[0391] Example prompt sentence:

[0392] Summarize the following text:

[0393] [Text extracted from PDF]

[0394] It should be around 100-200 characters in length, focus on the key points, and be concise.

[0395] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0396] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0397] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0398] [Second embodiment]

[0399] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0400] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0401] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0402] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0403] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0404] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0405] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0406] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0407] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0408] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0409] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0410] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0411] The present invention relates to a system that uses a generative AI model to summarize instruction manuals and convert them into video and audio formats. The purpose of this system is to provide users with instruction manuals in a format that is easy to understand. Specific embodiments of this system are described below.

[0412] User Operation

[0413] Users access the system to upload instruction manual files (PDF or text files). Users click the "Upload" button in their web browser or mobile application and select the instruction manual they want to upload from the file selection dialog.

[0414] Terminal operation

[0415] The device reads the instruction manual file selected by the user and sends the binary data to the server. Specifically, it sends the file to the server as an HTTP POST request, requesting that the server analyze the instruction manual.

[0416] Server Processing

[0417] The server receives the file sent from the terminal and analyzes its contents. For analysis, an appropriate library (e.g., a PDF analysis library or text analysis algorithm) is used. For PDF files, text extraction is performed, and for text files, the text data is read directly. Next, the plain text is preprocessed to remove unnecessary information and special characters.

[0418] Next, we use a generative AI model to summarize the parsed text. We use a generative AI model (e.g., GPT-4) to extract key points from the instruction manual and generate a concise summary. We evaluate the summary results and regenerate them if necessary.

[0419] Content Conversion and Delivery

[0420] The summarized text is then converted into video and audio formats. The server first inputs the summarized text into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. Then, video generation software is used to generate an animated video, which includes slides and visuals based on the summarized content. The generated audio file is also incorporated into the video.

[0421] Finally, the generated video and audio files are uploaded to cloud storage, and a download link or streaming URL is provided to the user's device, allowing the user to view the video and audio and understand the contents of the instruction manual.

[0422] Specific examples

[0423] For example, consider the case where a user wants to upload an instruction manual for a home appliance (e.g., a washing machine) to the system. The user accesses a web application and clicks the "Upload" button. The user selects a PDF file of the washing machine's instruction manual from the local storage and uploads it. The device then sends this PDF file to the server, which extracts text from the PDF and performs preprocessing.

[0424] The server uses a generative AI model to extract only the key information and generate a summary divided into sections such as "Initial Setup," "How to Use," and "Troubleshooting." Based on this summary, a text-to-speech engine is used to generate an audio file, and video generation software is used to create an animated video.

[0425] Finally, the server uploads the generated video and audio files to cloud storage and provides a download link to the user. The user can then use the provided link to watch the video and understand how to set up and use the washing machine. In this way, a system is realized that converts complex instruction manuals into a user-friendly format, facilitating user understanding.

[0426] The processing flow will be explained below.

[0427] Step 1:

[0428] The user starts uploading the instruction manual.

[0429] Users click the "Upload" button in their web browser or mobile application and select the instruction manual (PDF or text file) they want to upload from the file selection dialog.

[0430] Step 2:

[0431] The device sends the uploaded file to the server.

[0432] The terminal reads the selected file, converts it to binary data, and then generates an HTTP POST request and sends it to the server with the file data.

[0433] Step 3:

[0434] The server receives and parses the file.

[0435] The server receives the HTTP request and extracts the attached file data. For PDF files, the server extracts the text using a PDF parsing library. For text files, the server reads the text data directly. The extracted text data is then preprocessed to remove unnecessary whitespace and special characters.

[0436] Step 4:

[0437] The server generates the summary using the generative AI model.

[0438] The server loads the generative AI model and feeds it preprocessed text data. The AI ​​model extracts key information from this data and generates a concise summary. The summary is evaluated and regenerated if necessary.

[0439] Step 5:

[0440] The server converts the summary into video or audio format.

[0441] The server inputs the summarized text into a text-to-speech engine to generate an audio file, then uses video generation software to generate an animated video based on the summarized content, adding text slides and visuals to the video and incorporating the generated audio file.

[0442] Step 6:

[0443] The server provides the generated video and audio files to the user.

[0444] The server uploads the generated video and audio files to cloud storage and generates and sends a download link or streaming URL to the user's device.

[0445] Step 7:

[0446] View and listen to user-generated content.

[0447] Users can download or stream the video and audio using the provided link, and can pause, rewind, or fast-forward to the section they want while watching.

[0448] Through these steps, a system is realized that converts the contents of an instruction manual into a concise, easy-to-understand video or audio format and provides it to users.

[0449] Example 1

[0450] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0451] The instruction manuals that accompany a wide variety of modern products are often complex and require time and effort for users to understand. In particular, if they contain long texts or a lot of technical terminology, it can be difficult for users to understand, and they may not use or maintain the product properly. Therefore, there is a need for a method to communicate the contents of the instruction manual efficiently and effectively.

[0452] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0453] In this invention, the server includes a means for analyzing data of the instruction manual, a means for summarizing the analyzed data using a generative AI model, and a means for converting the summarized data into audio and video formats, thereby enabling the user to understand the contents of the instruction manual concisely and efficiently.

[0454] An "instruction manual" is a document that provides information on how to use a product or service, installation procedures, maintenance methods, troubleshooting, etc.

[0455] "Data" means content expressed in the form of information, instructions, numbers, etc., that is stored and transmitted electronically.

[0456] "Analysis" is the process of breaking down complex data and information and converting it into an understandable form.

[0457] A "generative AI model" is an artificial intelligence algorithm that uses deep learning and natural language processing to generate new data and text.

[0458] A "summary" is a concise summary of a large amount of information, extracting only the main points and shortening them.

[0459] "Speech" is the electronic representation of spoken words or other acoustic signals.

[0460] "Video" refers to the electronic representation of visual information, including text, images, animations, etc.

[0461] "User" means a person or entity that uses a system or product.

[0462] A "terminal" is an electronic device that allows a user to access and operate the system.

[0463] "Offering" is the act of giving something to someone else or making it available for their use.

[0464] The present invention relates to a system that uses a generative AI model to summarize instruction manuals and convert them into audio and video formats. The purpose of this system is to provide users with instruction manuals in a format that is easy to understand. Specific embodiments of this system are described below.

[0465] User Operation

[0466] Users access the system to upload instruction manual files (PDF or text files). They click the "Upload" button in their web browser or mobile application and select the instruction manual they want to upload from the file selection dialog. Once the upload is complete, the device prepares to send the file to the server.

[0467] Terminal operation

[0468] The device loads the instruction manual file selected by the user and sends the binary data to the server. Specifically, the device creates an HTTP POST request to send the selected file to the server as binary data. This request also includes metadata such as the file name and file type.

[0469] Server Processing

[0470] The server receives the file sent from the terminal. After receiving it, it identifies the file type and selects the appropriate parsing method. If it is a PDF file, it uses a PDF parsing library (e.g., PDFBox or PyPDF2) to extract the text data. If it is a text file, it reads the text data directly.

[0471] The server then preprocesses the extracted text data, removing unnecessary information (e.g., advertisements or specific product brands) and special characters, and checking the accuracy of grammar and syntax, correcting errors if necessary.

[0472] The server inputs the preprocessed text into a generative AI model (e.g., OpenAI's GPT-4) to generate a summary. Specifically, it invokes the generative AI model and gives it the following prompt:

[0473] Prompt: "Summarize the text from the instruction manual below. Extract the key points and summarize them concisely."

[0474] The generative AI model extracts important information based on the input text and generates a summary.

[0475] Content Conversion and Delivery

[0476] The server then converts the generated summary into audio and video formats. First, the summarized text is fed into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. Then, video generation software (e.g., Adobe After Effects or Apple Motion) is used to create an animated video based on the audio data. Slides and visuals based on the summarized content are added to the video.

[0477] Finally, the server uploads the generated video and audio files to cloud storage (e.g., Amazon S3 or Google Cloud Storage), obtains the file URL on the cloud storage, and provides a download link or streaming URL to the user's device.

[0478] Users can use the links to view videos and audio, which allows them to easily understand the contents of the instruction manual through both sight and sound.

[0479] Specific examples

[0480] For example, consider the case where a user wants to upload an instruction manual for a home appliance (e.g., a washing machine). The user accesses a web application, clicks the "Upload" button, selects the PDF file of the washing machine's instruction manual from the local storage, and uploads it. The device then sends this PDF file to the server, which extracts text from the PDF and performs preprocessing.

[0481] The server uses a generative AI model to extract only the key information and generate a summary divided into sections such as "Initial Setup," "How to Use," and "Troubleshooting." Based on this summary, a text-to-speech engine is used to generate an audio file, and video generation software is used to create an animated video.

[0482] Finally, the server uploads the generated video and audio files to cloud storage and provides a download link to the user. The user can then use the provided link to watch the video and understand how to set up and use the washing machine. This system makes it possible to provide complex instruction manuals in a concise, visually easy-to-understand format.

[0483] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0484] Step 1: User interaction

[0485] Users access the system and upload instruction manual files (PDF or text files). Specifically, they click the "Upload" button in their web browser or mobile application and select the instruction manual they want to upload from the file selection dialog. The input is the instruction manual file specified by the user, and the output is the binary data of the selected file.

[0486] Step 2: Loading and sending the file

[0487] The terminal reads the instruction manual file selected by the user and sends the binary data to the server. Specifically, the terminal creates an HTTP POST request and sends the selected file as binary data to the server. The input is the binary data of the file selected by the user, and the output is the HTTP POST request sent to the server.

[0488] Step 3: Receiving and analyzing the file

[0489] The server receives the file sent from the terminal. After receiving it, it identifies the file type and selects the appropriate parsing method. If it is a PDF file, it uses a PDF parsing library (e.g., PDFBox or PyPDF2) to extract the text data. If it is a text file, it reads the text data directly. The input is the file data received by the server, and the output is the extracted text data.

[0490] Step 4: Preprocessing the data

[0491] The server preprocesses the extracted text data. This preprocessing removes unnecessary information (e.g., advertisements or specific product brands) and special characters. It also checks the accuracy of grammar and syntax and corrects errors as needed. The input is the extracted text data, and the output is the preprocessed, clean text data.

[0492] Step 5: Summarizing the text

[0493] The server inputs the preprocessed text into a generative AI model (e.g., GPT-4) to generate a summary. Specifically, it invokes the generative AI model and gives it the following prompt:

[0494] Prompt: "Summarize the text from the instruction manual below. Extract the key points and summarize them concisely."

[0495] The input is preprocessed text data and a prompt sentence, and the output is the generated summary sentence.

[0496] Step 6: Converting the summary into audio and video formats

[0497] The server inputs the generated summary into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. It then uses video generation software (e.g., Adobe After Effects or Apple Motion) to create an animated video, adding slides and visuals based on the summarized content. The input is the summary, and the output is the generated audio and video files.

[0498] Step 7: Upload and provide files

[0499] The server uploads the generated video and audio files to cloud storage (e.g., Amazon S3 or Google Cloud Storage). It obtains the file URL on the cloud storage and provides a download link or streaming URL to the user's device. The input is the generated audio and video files, and the output is the download link or streaming URL provided to the user.

[0500] Specific examples

[0501] For example, if a user wants to upload a washing machine instruction manual, the actual operation steps are as follows:

[0502] Step 1: The user accesses the web application, clicks the "Upload" button, selects the washing machine instruction manual (PDF file), and clicks the "Submit" button.

[0503] Step 2: The device reads the selected PDF file and creates an HTTP POST request to send to the server.

[0504] Step 3: The server receives the file and uses PDFBox to extract and preprocess the text data.

[0505] Step 4: The server inputs the preprocessed text into the generative AI model to generate a summary.

[0506] Step 5: The server feeds the summary into Google Text-to-Speech to generate an audio file, and then uses Adobe After Effects to generate a video of the slides and audio synchronized to the text.

[0507] Step 6: The server uploads the generated video and audio files to Amazon S3 and provides a download link to the user.

[0508] Step 7: The user uses the provided link to watch and listen to videos and audio to understand how to set up and use the washing machine.

[0509] This series of processes provides the instruction manual in a concise and visually easy-to-understand format.

[0510] (Application example 1)

[0511] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0512] Conventional instruction manuals require a huge amount of time and effort for users to understand their contents, which is a major obstacle to efficient work, especially on factory floors. Furthermore, paper-based or simple text-based manuals make it difficult to visually understand procedures and operations, increasing the risk of operational errors and maintenance mistakes. This can lead to reduced production efficiency and safety, so a more user-friendly instruction manual provision system is needed.

[0513] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0514] In this invention, the server includes a means for analyzing the text of the instruction manual, a means for summarizing the analyzed text using a generative AI model, a means for converting the summarized text into video and audio formats, a means for providing the summarized content to a user via smart glasses in visual and audio formats, and a means for providing the converted content to a user terminal, thereby enabling the user to understand the instruction manual and maintenance procedures of a factory robot in real time through visual and audio formats, thereby achieving efficient and safe work.

[0515] The "means for analyzing the text of the instruction manual" has the function of reading the document data of the instruction manual uploaded by the user and analyzing its contents.

[0516] "Means for summarizing analyzed text using a generative AI model" refers to a function that has the function of concisely summarizing the contents of analyzed text data through a generative AI model that uses natural language processing technology.

[0517] A "means for converting summarized text into video and audio formats" is a means for using a text-to-speech engine or video generation software to create audio files and animated videos based on the summarized text.

[0518] The "means for visually and audibly providing summarized content to a user via smart glasses" means a means for providing summarized text content to the user's visual and auditory senses in real time via smart glasses.

[0519] The "means for providing converted content to a user terminal" refers to a function that uploads the generated video and audio content to cloud storage and provides a download link or streaming URL to the user terminal.

[0520] The "means for removing unnecessary information" has a function of performing preprocessing to remove unimportant information and special characters that become noise from the analyzed text data.

[0521] "Means for converting to an audio file using a text-to-speech engine" means a means for converting the summarized text into audio using text-to-speech technology and generating an audio file.

[0522] The "means of converting the text into an animated video using video generation software" has the function of generating an animated video based on the summarized text, along with slides and visuals.

[0523] The "means for providing the contents of the factory robot's instruction manual in real time" has the function of presenting information on the handling and maintenance of the factory robot to workers in real time through smart glasses.

[0524] MODE FOR CARRYING OUT THE INVENTION

[0525] The present invention relates to a system for providing instruction manuals in a format that is easy for users to understand. This system analyzes the text of the instruction manual, converts the summarized content into video and audio formats, and provides the content in real time via smart glasses. An embodiment of this system will be specifically described below.

[0526] Program Overview

[0527] The system has the following main functions:

[0528] 1. Ability to analyze instruction manual text

[0529] 2. Ability to summarize text using generative AI models

[0530] 3. Ability to convert summarized text into video and audio formats

[0531] 4. Function to provide content to smart glasses and user devices

[0532] Hardware and Software Configuration

[0533] The hardware and software used includes:

[0534] Terminal: The device to which the user uploads the instruction manual, such as a PC or smartphone.

[0535] Server: A central server that performs text analysis, summarization, and content conversion. It uses Python, a PDF analysis library, GPT-4, a Text-to-Speech API, and a video generation API as its main software.

[0536] Smart glasses: devices that provide real-time visual and audio information, such as Google Glass.

[0537] Data processing and calculation

[0538] 1. Upload and analyze the instruction manual:

[0539] Users upload a PDF of the instruction manual through their device, and the server extracts the text from the PDF and removes unnecessary information.

[0540] 2. Summary generation:

[0541] The extracted text is summarized using a generative AI model (GPT-4). As a concrete example, the following prompt sentence is sent to the AI ​​model to generate a summary:

[0542] Please summarize the contents of the following instruction manual:

[0543] "This robot requires daily inspection. The inspection procedure is as follows: remove part A, clean part B, and reinstall part C."

[0544] 3. Content Transformation:

[0545] The summarized text is converted into audio using a Text-to-Speech API, and then an animated video is generated using a Video Generation API.

[0546] 4. Provision to smart glasses and user devices:

[0547] The generated audio and video content is uploaded to cloud storage and provided to users in real time via smart glasses, and a download link or streaming URL is provided to the user's device as needed.

[0548] Specific examples

[0549] For example, when a factory worker wearing smart glasses performs maintenance on a robot, he or she uploads a PDF instruction manual to the system. The server extracts text from the PDF and generates a summary using GPT-4. The summary is then converted into a video with audio and animation using the Text-to-Speech API and the Video Generation API. This audio and video is then provided to the worker in real time via the smart glasses. This allows the worker to visually confirm the procedure while receiving audio instructions.

[0550] This system will enable factory workers to significantly reduce the time it takes to understand complex instruction manuals, enabling them to work efficiently and safely.

[0551] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0552] Step 1:

[0553] A user accesses the web or mobile application. The user clicks the "Upload" button, selects the instruction manual file (PDF or text file), and uploads it.

[0554] Input: Instruction manual file

[0555] Output: Binary data of the instruction manual

[0556] Specific operation: The user selects the desired instruction manual from the file selection dialog and presses the upload button. This action sends the binary data of the instruction manual to the terminal.

[0557] Step 2:

[0558] The terminal transmits the binary data of the instruction manual selected by the user to the server.

[0559] Input: Binary data of instruction manual

[0560] Output: Data sent to the server

[0561] Specific operation: The device sends the binary data of the instruction manual to the server as an HTTP POST request. This request uploads the file to the server.

[0562] Step 3:

[0563] The server analyzes the received instruction manual file and extracts the text data. If it is a PDF file, it uses a PDF analysis library to extract the text, and if it is a text file, it reads it directly.

[0564] Input: Binary data of the instruction manual uploaded to the server

[0565] Output: Extracted text data

[0566] What it does: The server uses a PDF parsing library to extract text from PDF files, or if it's a text file, reads the text data directly.

[0567] Step 4:

[0568] The server preprocesses the extracted text data to remove unnecessary information and special characters.

[0569] Input: Extracted text data

[0570] Output: Preprocessed clean text data

[0571] Specific operation: The server cleanses the text data using regular expressions and string manipulation to remove unnecessary information and special characters.

[0572] Step 5:

[0573] The server inputs the preprocessed text data into a generative AI model (GPT-4) to generate a summary.

[0574] Input: Preprocessed clean text data

[0575] Output: Summary text

[0576] Specific operation: The server inputs the following prompt to the generated AI model:

[0577] Please summarize the contents of the following instruction manual:

[0578] "This is an instruction manual divided into sections for setup, usage, and troubleshooting. The setup is as follows..."

[0579] A generative AI model generates a summary based on this prompt.

[0580] Step 6:

[0581] The server converts the generated summary text into audio using a Text-to-Speech API.

[0582] Input: Summary text

[0583] Output: Audio file (mp3, etc.)

[0584] Specific operation: The server sends the summary text to the Text-to-Speech API and receives the generated audio file.

[0585] Step 7:

[0586] The server generates animated videos based on the summary text using a video generation API.

[0587] Input: Summary text

[0588] Output: Animated video file

[0589] Specific operation: The server sends the summary text to the video generation API and receives the generated video file.

[0590] Step 8:

[0591] The server uploads the generated audio and video files to cloud storage and generates a link to provide to the user terminal and smart glasses.

[0592] Input: Audio and video files

[0593] Output: Upload and download links to cloud storage, streaming URL

[0594] Specific operation: The server uploads the audio and video files to a cloud storage service and provides the generated link to the user's application and smart glasses app.

[0595] Step 9:

[0596] Users can view and listen to the contents of the instruction manual summarized in real time through smart glasses and devices.

[0597] Input: Download link / Streaming URL

[0598] Output: Real-time viewing and listening content

[0599] What it does: The user clicks on the provided link or accesses the viewing screen with smart glasses to view audio and video content in real time.

[0600] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0601] The present invention is a system that not only summarizes instruction manuals using a generative AI model and converts the summaries into video and audio formats, but also combines it with an emotion engine that recognizes user emotions to provide more user-friendly content. Specific embodiments of this system are described below.

[0602] User Operation

[0603] First, users access the system and upload the instruction manual file (PDF or text file). Users click the "Upload" button in their web browser or mobile application and select the instruction manual they want to upload from the file selection dialog.

[0604] Terminal operation

[0605] The device reads the instruction manual file selected by the user, converts it into binary data, and then sends the file data to the server as an HTTP POST request to request analysis.

[0606] Server Processing

[0607] The server receives the file sent from the device and parses its contents. This involves extracting text from the PDF file using a PDF parsing library, or reading the text directly if it is a text file. The server then preprocesses the plain text, removing unnecessary whitespace and special characters.

[0608] The parsed text is then summarized using a generative AI model (e.g., GPT-4) to extract key points from the instruction manual and generate a concise summary. The final summary is evaluated and regenerated if necessary.

[0609] Content Conversion and Delivery

[0610] The summarized text is then converted into video or audio format. The server first inputs the summarized text into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. Then, video generation software is used to generate an animated video based on the summarized content. This video includes text slides and visuals, and also integrates the generated audio file.

[0611] Incorporating an emotion engine

[0612] The emotion engine recognizes the user's emotional state in real time and adjusts the content of the summary text and the tone and speed of the generated video and audio based on its output. For example, if the user feels confused or anxious about the information, the server will provide more detailed summary content at a slower pace based on the output of the emotion engine. The data from the emotion engine will also be used to adjust the animation and audio tone to reduce the user's psychological burden.

[0613] Content Submissions and Feedback

[0614] Finally, the server uploads the generated video and audio files to cloud storage and generates and sends a download link or streaming URL to the user's device. The user uses this link to watch the video or audio. The emotion engine monitors the user's reactions in real time while watching and adjusts the content as needed.

[0615] Specific examples

[0616] For example, consider a case where a user uploads a user manual for a home appliance (e.g., a smartphone). The user accesses a web application, clicks the "Upload" button, and selects the PDF file of the smartphone's user manual. The device then sends this file to a server, which extracts text from the PDF and uses a generative AI model to generate a summary of key sections.

[0617] The server then inputs the summarized text into a text-to-speech engine to generate an audio file, and uses video generation software to create an animated video. During this process, an emotion engine recognizes the user's emotions and adjusts the summary and video tone and speed as needed. For example, if the user is confused, the explanation will be more detailed and the video will be slower.

[0618] Finally, the server uploads the generated video and audio files to cloud storage and provides a download link to the user. The user clicks on this link to watch the video and understand how to set up the smartphone and use its functions. In this way, the present invention is a system that converts complex instruction manuals into a user-friendly format and provides them adaptively using an emotion engine.

[0619] The processing flow will be explained below.

[0620] Step 1:

[0621] The user starts uploading the instruction manual.

[0622] Users click the "Upload" button in their web browser or mobile application and select the instruction manual (PDF or text file) they want to upload from the file selection dialog.

[0623] Step 2:

[0624] The device sends the uploaded file to the server.

[0625] The terminal reads the selected file, converts it to binary data, then generates an HTTP POST request and sends it to the server with the file data.

[0626] Step 3:

[0627] The server receives and parses the file.

[0628] The server receives the HTTP request and extracts the attachment data. For PDF files, the server extracts the text using a PDF parsing library. For text files, the server reads the text data directly. The extracted text data is then preprocessed to remove unnecessary whitespace and special characters.

[0629] Step 4:

[0630] The server generates the summary using the generative AI model.

[0631] The server loads the generative AI model and feeds it preprocessed text data. The AI ​​model extracts key information from this data and generates a concise summary. The generated summary is evaluated and regenerated if necessary.

[0632] Step 5:

[0633] The server uses an emotion engine to recognize the user's emotional state.

[0634] The server uses an emotion engine to collect and analyze the user's real-time emotion data, such as facial expressions, tone of voice, and eye movements, to understand the user's emotional state.

[0635] Step 6:

[0636] The server converts the summary into video or audio format.

[0637] The server inputs the summarized text into a text-to-speech engine to generate an audio file, and then uses video generation software to generate an animated video based on the summarized content, adjusting the tone and speed of the video and audio accordingly based on data from the emotion engine.

[0638] Step 7:

[0639] The server provides the generated video and audio files to the user.

[0640] The server uploads the generated video and audio files to cloud storage and generates and sends a download link or streaming URL to the user's device.

[0641] Step 8:

[0642] View and listen to user-generated content.

[0643] Users can download or stream the video and audio using the provided link, and can pause, rewind, or fast-forward to the section they want while watching.

[0644] Step 9:

[0645] The server continuously monitors the user's emotional state.

[0646] The server monitors the user's emotional state in real time while watching video or listening to audio, collecting feedback and adjusting or regenerating content as needed based on this data.

[0647] This makes it possible to realize a system that provides the contents of the instruction manual to the user in an optimal format and makes it easy to understand.

[0648] Example 2

[0649] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0650] Conventional instruction manuals are often complex and difficult to understand, making it difficult for users to quickly obtain the information they need. Furthermore, conventional technologies provide information in a uniform format without considering the user's emotional state, making it difficult to provide a user-friendly experience. Furthermore, technologies for converting instruction manual summaries into video or audio formats are limited, making it difficult to meet the diverse needs of users.

[0651] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for analyzing the instruction manual document, means for summarizing the analyzed document using a generative AI model, means for converting the summarized document into video and audio format, means for uploading the converted content to cloud storage to be provided to the user terminal, and means for recognizing the user's emotional state using an emotion engine and adjusting the summary content and the tone and speed of the content. This not only enables the user to quickly obtain the information they need, but also provides the information in an appropriate format and at an appropriate pace depending on the user's emotional state, resulting in a more user-friendly experience.

[0652] An "instruction manual" is a document provided to explain how to use or operate a product or service.

[0653] A "document" is a collection of text or information written to convey a specific message.

[0654] "Analysis" is the process of examining data or information in detail to understand its structure and meaning.

[0655] A "generative AI model" is a model that uses artificial intelligence technology to generate new text or data.

[0656] A "summary" is a concise summary of the important parts of a document or data.

[0657] A "video format" is a data format that expresses movement by playing back a series of images or videos over time.

[0658] An "audio format" is a format in which sound or voice is represented as digital data.

[0659] "Cloud storage" is a remote data storage service that allows you to store and access data over the Internet.

[0660] An "emotion engine" is a technology / system that analyzes a user's emotional state in real time and utilizes the results.

[0661] A "user terminal" is hardware or a device that a user directly operates to input or receive information.

[0662] "Tone" refers to the tone, timbre, and atmosphere of a voice or text.

[0663] "Speed" refers to the speed at which information or content is provided to users.

[0664] The present invention is a system for analyzing an instruction manual, summarizing the contents thereof, and providing the summarization to a user. A specific embodiment of this system will be described below.

[0665] System Overview

[0666] Hardware Configuration

[0667] This system consists of three main components: user terminals, servers, and cloud storage. User terminals are devices such as PCs, smartphones, and tablets, while servers are computing devices for analysis and generation processing. Cloud storage is a remote data storage service that stores generated content and provides it to users.

[0668] Software Configuration

[0669] The system includes a PDF parsing library, a generative AI model (e.g., GPT-4), a text-to-speech engine (e.g., Google Text-to-Speech), video generation software, and an emotion engine.

[0670] Processing flow

[0671] User Operation

[0672] Users use a web browser or mobile application to upload instruction manual files (PDF or text files) to the system. Specifically, they access the system, click the "Upload" button, and select the instruction manual they want to upload from the file selection dialog.

[0673] Terminal operation

[0674] The device reads the instruction manual file selected by the user, converts it into binary data, and then sends the file data to the server as an HTTP POST request.

[0675] Server Processing

[0676] The server receives the file sent from the device and analyzes its contents. For PDF files, it extracts the text using a PDF analysis library, while for text files, it reads the text data directly. It then summarizes the analyzed text using a generative AI model. The generative AI model used (e.g., GPT-4) is given a prompt such as "Please briefly summarize the key points of this instruction manual" to generate an appropriate summary.

[0677] Content generation

[0678] The server inputs the summarized text into a text-to-speech engine to generate an audio file, and then uses video generation software to generate an animated video based on the summarized content, incorporating text slides and visuals and the generated audio file.

[0679] Emotion Engine Operation

[0680] The emotion engine recognizes the user's emotional state in real time. For example, if the user is using a webcam, it uses facial expression recognition technology to analyze their emotions and sends the results to the server. Based on the results, the server adjusts the content of the summary text and the tone and speed of the generated content.

[0681] Content provision

[0682] The server uploads the generated video and audio files to cloud storage and provides a download link or streaming URL to the user's device. The user uses this link to watch the video or audio. During viewing, the emotion engine monitors the user's reactions in real time and adjusts the content as needed.

[0683] Specific examples

[0684] For example, consider a user uploading a user manual for a home appliance (e.g., a smartphone). The user accesses a web application, clicks the "Upload" button, and selects the PDF file of the smartphone's user manual. The device sends this file to a server, which extracts text from the PDF and uses a generative AI model to generate a summary of key sections. The server then inputs the summarized text into a text-to-speech engine to generate an audio file. It then uses video generation software to create an animated video. During this process, an emotion engine recognizes the user's emotions and adjusts the summary content, tone, and speed of the video as needed. Finally, the server uploads the generated video and audio files to cloud storage and provides the user with a download link. The user clicks the link to watch the video and learn how to set up the smartphone and use its features.

[0685] Prompt Sentence Examples

[0686] "Please briefly summarize the important points of this instruction manual."

[0687] The above is a detailed description of the embodiment of the present invention. This system can convert complex instruction manuals into a user-friendly format and provide them adaptively using an emotion engine.

[0688] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0689] Step 1:

[0690] The user uploads an instruction manual to the system. The user accesses the system, clicks the "Upload" button using a web browser or mobile application, and selects the instruction manual file (PDF or text file) from the file selection dialog. The input is the instruction manual file, and the output is the path to the selected file.

[0691] Step 2:

[0692] The terminal reads the selected file and converts it to binary data. The terminal uses the file system API to read the specified file and converts it to binary data. It then sends this binary data to the server as an HTTP POST request. The input is the path of the file selected by the user, and the output is the binary data sent to the server.

[0693] Step 3:

[0694] The server receives the binary data sent from the terminal, saves the file, and analyzes the contents. The server first saves the file in a temporary directory. Next, if it is a PDF file, it uses a PDF parsing library to extract the text, or if it is a text file, it reads the text data directly. The input is binary data, and the output is the extracted plain text. Specific operations include calling the API of the PDF parsing library to extract the text from the file.

[0695] Step 4:

[0696] The server summarizes the extracted text using a generative AI model. The server preprocesses the extracted plain text to remove unnecessary white space and special characters. The generative AI model is then given a prompt sentence, "Please briefly summarize the important points of this instruction manual," to generate summarized text. The input is the parsed plain text, and the output is the summarized text.

[0697] Step 5:

[0698] The server converts the summarized text into audio format. The server inputs the summarized text into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. The input is the summarized text, and the output is an audio file. Specifically, the Google Text-to-Speech API is used to convert text into audio.

[0699] Step 6:

[0700] The server generates a video that includes an audio file. The server uses video generation software to create an animated video by integrating text slides and visuals with the generated audio file. The input is the summarized text and audio file, and the output is the generated video file. Specifically, the video generation software runs a script to generate the video.

[0701] Step 7:

[0702] The emotion engine recognizes the user's emotional state in real time. If the user is using a webcam, the emotion engine uses facial expression recognition technology to analyze emotions and sends the results to the server. The input is the video feed from the webcam, and the output is the analyzed emotion data. Specifically, it runs a facial expression recognition algorithm to analyze the user's emotions.

[0703] Step 8:

[0704] The server adjusts the content based on the output of the emotion engine. The server uses the emotion data to adjust the content of the summary text and the tone and speed of the generated video and audio. The input is emotion data, and the output is the adjusted content. Specifically, the server changes the parameters of the generation algorithm in real time.

[0705] Step 9:

[0706] The server uploads the generated video and audio files to cloud storage. The server uploads the files using a cloud storage service and provides an access link to the user's device. The input is the generated content file, and the output is the upload result to cloud storage and the access link. Specifically, the server uses the cloud storage API to upload the files and generate a link.

[0707] Step 10:

[0708] The user views the content using the provided link. The user clicks on a download link or streaming URL to view the video or audio. The input is the access link, and the output is the content to be viewed. The specific operation is to play the content using a browser or media player.

[0709] The above is a detailed flow of the program processing for this system. The input and output at each step are clearly stated, and the type of data processing and calculation that is performed is specifically explained.

[0710] (Application example 2)

[0711] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0712] In recent years, many factories have introduced new robots, but complicated instruction manuals hinder workers' quick understanding, resulting in a decline in work efficiency. Furthermore, there is a lack of technology to adapt instructions according to the user's emotional state, which often leaves workers confused.

[0713] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing the text of the instruction manual, means for summarizing the analyzed text using a generative AI model, means for converting the summarized text into video and audio format, means for providing the converted content to a user terminal, means for recognizing the user's emotional state in real time, and means for adjusting the content of the summarized text and the tone and speed of the video and audio based on the user's emotional state. This makes it possible to improve work efficiency and reduce user confusion.

[0714] An "instruction manual" is a document that contains information on how to use, maintain, and handle a product.

[0715] "Analyzing" means breaking down given data and examining each element in detail.

[0716] A "generative AI model" is a model that uses artificial intelligence techniques to generate new data, such as summarizing or automatically generating text.

[0717] To "summarize" means to summarize a longer piece of text or information in a concise form.

[0718] "Video format" is a format that expresses information as moving images.

[0719] "Audio format" is a format in which information is conveyed auditorily.

[0720] A "user terminal" is a device such as a computer or smartphone that is directly operated by a user.

[0721] "Emotional state" refers to a user's current psychological state or mood.

[0722] "Real-time" is a term that refers to processing or response occurring immediately, as in actual time.

[0723] "Tone" is an expression that refers to the tone or color of a sound or voice.

[0724] "Speed" refers to the speed at which information is transmitted within a given period of time.

[0725] This system summarizes the complex content of instruction manuals, converts them into video and audio formats, and adaptively adjusts the content according to the user's emotions, allowing factory workers to easily understand the instruction manuals for newly installed robots and improving work efficiency.

[0726] System configuration

[0727] 1. User Operation

[0728] First, the user uploads the instruction manual (PDF or text file) using the smartphone app. Click the "Upload" button in the app and select the file to upload from the file selection dialog.

[0729] 2. Terminal operation

[0730] The uploaded file is read by the smartphone and converted into binary data, which is then sent to the server as an HTTP POST request.

[0731] 3. Server Processing

[0732] The server uses a PDF parsing library (e.g., fitz) to extract text from the PDF file, or reads it directly in the case of text files.

[0733] Then, a generative AI model (e.g., GPT-4) is used to extract key points from the plain text and generate a concise summary. A text-to-speech engine (e.g., gTTS) is then used to generate an audio file, which is then used to create an animated video using video generation software (e.g., moviepy).

[0734] 4. Emotion engine integration

[0735] The emotion engine recognizes the user's emotional state in real time through the user's camera feed. For example, if the user is confused, the server will adjust the summary, tone and speed of the video to provide a more detailed and slower explanation.

[0736] 5. Content Submissions and Feedback

[0737] The generated video and audio files are uploaded to cloud storage, and a download link or streaming URL is generated for the user. The user can use the link to watch the video or audio. The emotion engine monitors the user's reactions in real time while watching and adjusts the content as needed.

[0738] Specific examples

[0739] For example, a factory worker uploads a PDF instruction manual for a new robot to the app. The server extracts the text from the PDF and uses a generative AI model to summarize the key points of the manual. The summarized text is then fed into a text-to-speech engine to generate an audio file, which is then used by video generation software to create an animated video.

[0740] The emotion engine monitors the user's emotional state in real time and regenerates the summary content if they are confused, providing more detailed and slower explanations.

[0741] Prompt Sentence Examples

[0742] Summarize the following text:

[0743] [Text extracted from PDF]

[0744] It should be around 100-200 characters in length, focus on the key points, and be concise.

[0745] As described above, this system contributes to improving work efficiency by converting complex instruction manuals into an easily understandable format and providing adaptive explanations using an emotion engine.

[0746] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0747] Step 1:

[0748] The user uploads an instruction manual (PDF or text file) using a smartphone app. The input data is the instruction manual file on the user's device, and the output data is binary data sent to the server.

[0749] Step 2:

[0750] The terminal reads the uploaded file, converts it to binary data, and sends the converted binary data to the server as an HTTP POST request. The input is the file selected by the user, and the output is the binary data sent to the server.

[0751] Step 3:

[0752] The server parses the received binary data and extracts the text from the PDF file using a PDF parsing library (e.g., fitz) or reads the data directly from the text file. The input is the binary data sent to the server, and the output is the extracted text data.

[0753] Step 4:

[0754] The server generates a summary from the plain text using a generative AI model (e.g., GPT-4). In this process, the input is the extracted text data, and the output is the generated summary text. Specifically, a prompt sentence is input to the generative AI model, and the model performs a process of summarizing the key points.

[0755] Step 5:

[0756] The server inputs the generated summary text into a text-to-speech engine (e.g., gTTS) to generate an audio file. The input is the summarized text data, and the output is an audio file.

[0757] Step 6:

[0758] The server creates an animated video using video generation software (e.g., moviepy) based on the generated audio file. The input is the audio file and summary text, and the output is the generated animated video file.

[0759] Step 7:

[0760] The emotion engine recognizes the user's emotional state in real time through the user's camera feed. The input is data from the user's camera feed, and the output is data on the user's emotional state. Based on the emotional state, it adjusts the tone and speed of the video and audio.

[0761] Step 8:

[0762] The server realigns the summary text and the generated video and audio files according to the output data of the emotion engine. The input is the emotion engine data, and the output is the realigned content.

[0763] Step 9:

[0764] Finally, the generated video and audio files are uploaded to cloud storage, and a download link or streaming URL is provided to the user's device. The input is the generated video and audio files, and the output is uploading them to cloud storage and providing a link to the user.

[0765] A concrete example would be a factory floor worker uploading a new robot instruction manual to an app, watching abridged video and audio, and receiving detailed, slow-fire explanations when the emotion engine detects confusion.

[0766] Example prompt sentence:

[0767] Summarize the following text:

[0768] [Text extracted from PDF]

[0769] It should be around 100-200 characters in length, focus on the key points, and be concise.

[0770] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0771] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0772] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0773] [Third embodiment]

[0774] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0775] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0776] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0777] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0778] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0779] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0780] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0781] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0782] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0783] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0784] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0785] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0786] The present invention relates to a system that uses a generative AI model to summarize instruction manuals and convert them into video and audio formats. The purpose of this system is to provide users with instruction manuals in a format that is easy to understand. Specific embodiments of this system are described below.

[0787] User Operation

[0788] Users access the system to upload instruction manual files (PDF or text files). Users click the "Upload" button in their web browser or mobile application and select the instruction manual they want to upload from the file selection dialog.

[0789] Terminal operation

[0790] The device reads the instruction manual file selected by the user and sends the binary data to the server. Specifically, it sends the file to the server as an HTTP POST request, requesting that the server analyze the instruction manual.

[0791] Server Processing

[0792] The server receives the file sent from the terminal and analyzes its contents. For analysis, an appropriate library (e.g., a PDF analysis library or text analysis algorithm) is used. For PDF files, text extraction is performed, and for text files, the text data is read directly. Next, the plain text is preprocessed to remove unnecessary information and special characters.

[0793] Next, we use a generative AI model to summarize the parsed text. We use a generative AI model (e.g., GPT-4) to extract key points from the instruction manual and generate a concise summary. We evaluate the summary results and regenerate them if necessary.

[0794] Content Conversion and Delivery

[0795] The summarized text is then converted into video and audio formats. The server first inputs the summarized text into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. Then, video generation software is used to generate an animated video, which includes slides and visuals based on the summarized content. The generated audio file is also incorporated into the video.

[0796] Finally, the generated video and audio files are uploaded to cloud storage, and a download link or streaming URL is provided to the user's device, allowing the user to view the video and audio and understand the contents of the instruction manual.

[0797] Specific examples

[0798] For example, consider the case where a user wants to upload an instruction manual for a home appliance (e.g., a washing machine) to the system. The user accesses a web application and clicks the "Upload" button. The user selects a PDF file of the washing machine's instruction manual from the local storage and uploads it. The device then sends this PDF file to the server, which extracts text from the PDF and performs preprocessing.

[0799] The server uses a generative AI model to extract only the key information and generate a summary divided into sections such as "Initial Setup," "How to Use," and "Troubleshooting." Based on this summary, a text-to-speech engine is used to generate an audio file, and video generation software is used to create an animated video.

[0800] Finally, the server uploads the generated video and audio files to cloud storage and provides a download link to the user. The user can then use the provided link to watch the video and understand how to set up and use the washing machine. In this way, a system is realized that converts complex instruction manuals into a user-friendly format, facilitating user understanding.

[0801] The processing flow will be explained below.

[0802] Step 1:

[0803] The user starts uploading the instruction manual.

[0804] Users click the "Upload" button in their web browser or mobile application and select the instruction manual (PDF or text file) they want to upload from the file selection dialog.

[0805] Step 2:

[0806] The device sends the uploaded file to the server.

[0807] The terminal reads the selected file, converts it to binary data, and then generates an HTTP POST request and sends it to the server with the file data.

[0808] Step 3:

[0809] The server receives and parses the file.

[0810] The server receives the HTTP request and extracts the attached file data. For PDF files, the server extracts the text using a PDF parsing library. For text files, the server reads the text data directly. The extracted text data is then preprocessed to remove unnecessary whitespace and special characters.

[0811] Step 4:

[0812] The server generates the summary using the generative AI model.

[0813] The server loads the generative AI model and feeds it preprocessed text data. The AI ​​model extracts key information from this data and generates a concise summary. The summary is evaluated and regenerated if necessary.

[0814] Step 5:

[0815] The server converts the summary into video or audio format.

[0816] The server inputs the summarized text into a text-to-speech engine to generate an audio file, then uses video generation software to generate an animated video based on the summarized content, adding text slides and visuals to the video and incorporating the generated audio file.

[0817] Step 6:

[0818] The server provides the generated video and audio files to the user.

[0819] The server uploads the generated video and audio files to cloud storage and generates and sends a download link or streaming URL to the user's device.

[0820] Step 7:

[0821] View and listen to user-generated content.

[0822] Users can download or stream the video and audio using the provided link, and can pause, rewind, or fast-forward to the section they want while watching.

[0823] Through these steps, a system is realized that converts the contents of an instruction manual into a concise, easy-to-understand video or audio format and provides it to users.

[0824] Example 1

[0825] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0826] The instruction manuals that accompany a wide variety of modern products are often complex and require time and effort for users to understand. In particular, if they contain long texts or a lot of technical terminology, it can be difficult for users to understand, and they may not use or maintain the product properly. Therefore, there is a need for a method to communicate the contents of the instruction manual efficiently and effectively.

[0827] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0828] In this invention, the server includes a means for analyzing data of the instruction manual, a means for summarizing the analyzed data using a generative AI model, and a means for converting the summarized data into audio and video formats, thereby enabling the user to understand the contents of the instruction manual concisely and efficiently.

[0829] An "instruction manual" is a document that provides information on how to use a product or service, installation procedures, maintenance methods, troubleshooting, etc.

[0830] "Data" means content expressed in the form of information, instructions, numbers, etc., that is stored and transmitted electronically.

[0831] "Analysis" is the process of breaking down complex data and information and converting it into an understandable form.

[0832] A "generative AI model" is an artificial intelligence algorithm that uses deep learning and natural language processing to generate new data and text.

[0833] A "summary" is a concise summary of a large amount of information, extracting only the main points and shortening them.

[0834] "Speech" is the electronic representation of spoken words or other acoustic signals.

[0835] "Video" refers to the electronic representation of visual information, including text, images, animations, etc.

[0836] "User" means a person or entity that uses a system or product.

[0837] A "terminal" is an electronic device that allows a user to access and operate the system.

[0838] "Offering" is the act of giving something to someone else or making it available for their use.

[0839] The present invention relates to a system that uses a generative AI model to summarize instruction manuals and convert them into audio and video formats. The purpose of this system is to provide users with instruction manuals in a format that is easy to understand. Specific embodiments of this system are described below.

[0840] User Operation

[0841] Users access the system to upload instruction manual files (PDF or text files). They click the "Upload" button in their web browser or mobile application and select the instruction manual they want to upload from the file selection dialog. Once the upload is complete, the device prepares to send the file to the server.

[0842] Terminal operation

[0843] The device loads the instruction manual file selected by the user and sends the binary data to the server. Specifically, the device creates an HTTP POST request to send the selected file to the server as binary data. This request also includes metadata such as the file name and file type.

[0844] Server Processing

[0845] The server receives the file sent from the terminal. After receiving it, it identifies the file type and selects the appropriate parsing method. If it is a PDF file, it uses a PDF parsing library (e.g., PDFBox or PyPDF2) to extract the text data. If it is a text file, it reads the text data directly.

[0846] The server then preprocesses the extracted text data, removing unnecessary information (e.g., advertisements or specific product brands) and special characters, and checking the accuracy of grammar and syntax, correcting errors if necessary.

[0847] The server inputs the preprocessed text into a generative AI model (e.g., OpenAI's GPT-4) to generate a summary. Specifically, it invokes the generative AI model and gives it the following prompt:

[0848] Prompt: "Summarize the text from the instruction manual below. Extract the key points and summarize them concisely."

[0849] The generative AI model extracts important information based on the input text and generates a summary.

[0850] Content Conversion and Delivery

[0851] The server then converts the generated summary into audio and video formats. First, the summarized text is fed into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. Then, video generation software (e.g., Adobe After Effects or Apple Motion) is used to create an animated video based on the audio data. Slides and visuals based on the summarized content are added to the video.

[0852] Finally, the server uploads the generated video and audio files to cloud storage (e.g., Amazon S3 or Google Cloud Storage), obtains the file URL on the cloud storage, and provides a download link or streaming URL to the user's device.

[0853] Users can use the links to view videos and audio, which allows them to easily understand the contents of the instruction manual through both sight and sound.

[0854] Specific examples

[0855] For example, consider the case where a user wants to upload an instruction manual for a home appliance (e.g., a washing machine). The user accesses a web application, clicks the "Upload" button, selects the PDF file of the washing machine's instruction manual from the local storage, and uploads it. The device then sends this PDF file to the server, which extracts text from the PDF and performs preprocessing.

[0856] The server uses a generative AI model to extract only the key information and generate a summary divided into sections such as "Initial Setup," "How to Use," and "Troubleshooting." Based on this summary, a text-to-speech engine is used to generate an audio file, and video generation software is used to create an animated video.

[0857] Finally, the server uploads the generated video and audio files to cloud storage and provides a download link to the user. The user can then use the provided link to watch the video and understand how to set up and use the washing machine. This system makes it possible to provide complex instruction manuals in a concise, visually easy-to-understand format.

[0858] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0859] Step 1: User interaction

[0860] Users access the system and upload instruction manual files (PDF or text files). Specifically, they click the "Upload" button in their web browser or mobile application and select the instruction manual they want to upload from the file selection dialog. The input is the instruction manual file specified by the user, and the output is the binary data of the selected file.

[0861] Step 2: Loading and sending the file

[0862] The terminal reads the instruction manual file selected by the user and sends the binary data to the server. Specifically, the terminal creates an HTTP POST request and sends the selected file as binary data to the server. The input is the binary data of the file selected by the user, and the output is the HTTP POST request sent to the server.

[0863] Step 3: Receiving and analyzing the file

[0864] The server receives the file sent from the terminal. After receiving it, it identifies the file type and selects the appropriate parsing method. If it is a PDF file, it uses a PDF parsing library (e.g., PDFBox or PyPDF2) to extract the text data. If it is a text file, it reads the text data directly. The input is the file data received by the server, and the output is the extracted text data.

[0865] Step 4: Preprocessing the data

[0866] The server preprocesses the extracted text data. This preprocessing removes unnecessary information (e.g., advertisements or specific product brands) and special characters. It also checks the accuracy of grammar and syntax and corrects errors as needed. The input is the extracted text data, and the output is the preprocessed, clean text data.

[0867] Step 5: Summarizing the text

[0868] The server inputs the preprocessed text into a generative AI model (e.g., GPT-4) to generate a summary. Specifically, it invokes the generative AI model and gives it the following prompt:

[0869] Prompt: "Summarize the text from the instruction manual below. Extract the key points and summarize them concisely."

[0870] The input is preprocessed text data and a prompt sentence, and the output is the generated summary sentence.

[0871] Step 6: Converting the summary into audio and video formats

[0872] The server inputs the generated summary into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. It then uses video generation software (e.g., Adobe After Effects or Apple Motion) to create an animated video, adding slides and visuals based on the summarized content. The input is the summary, and the output is the generated audio and video files.

[0873] Step 7: Upload and provide files

[0874] The server uploads the generated video and audio files to cloud storage (e.g., Amazon S3 or Google Cloud Storage). It obtains the file URL on the cloud storage and provides a download link or streaming URL to the user's device. The input is the generated audio and video files, and the output is the download link or streaming URL provided to the user.

[0875] Specific examples

[0876] For example, if a user wants to upload a washing machine instruction manual, the actual operation steps are as follows:

[0877] Step 1: The user accesses the web application, clicks the "Upload" button, selects the washing machine instruction manual (PDF file), and clicks the "Submit" button.

[0878] Step 2: The device reads the selected PDF file and creates an HTTP POST request to send to the server.

[0879] Step 3: The server receives the file and uses PDFBox to extract and preprocess the text data.

[0880] Step 4: The server inputs the preprocessed text into the generative AI model to generate a summary.

[0881] Step 5: The server feeds the summary into Google Text-to-Speech to generate an audio file, and then uses Adobe After Effects to generate a video of the slides and audio synchronized to the text.

[0882] Step 6: The server uploads the generated video and audio files to Amazon S3 and provides a download link to the user.

[0883] Step 7: The user uses the provided link to watch and listen to videos and audio to understand how to set up and use the washing machine.

[0884] This series of processes provides the instruction manual in a concise and visually easy-to-understand format.

[0885] (Application example 1)

[0886] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0887] Conventional instruction manuals require a huge amount of time and effort for users to understand their contents, which is a major obstacle to efficient work, especially on factory floors. Furthermore, paper-based or simple text-based manuals make it difficult to visually understand procedures and operations, increasing the risk of operational errors and maintenance mistakes. This can lead to reduced production efficiency and safety, so a more user-friendly instruction manual provision system is needed.

[0888] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0889] In this invention, the server includes a means for analyzing the text of the instruction manual, a means for summarizing the analyzed text using a generative AI model, a means for converting the summarized text into video and audio formats, a means for providing the summarized content to a user via smart glasses in visual and audio formats, and a means for providing the converted content to a user terminal, thereby enabling the user to understand the instruction manual and maintenance procedures of a factory robot in real time through visual and audio formats, thereby achieving efficient and safe work.

[0890] The "means for analyzing the text of the instruction manual" has the function of reading the document data of the instruction manual uploaded by the user and analyzing its contents.

[0891] "Means for summarizing analyzed text using a generative AI model" refers to a function that has the function of concisely summarizing the contents of analyzed text data through a generative AI model that uses natural language processing technology.

[0892] A "means for converting summarized text into video and audio formats" is a means for using a text-to-speech engine or video generation software to create audio files and animated videos based on the summarized text.

[0893] The "means for visually and audibly providing summarized content to a user via smart glasses" means a means for providing summarized text content to the user's visual and auditory senses in real time via smart glasses.

[0894] The "means for providing converted content to a user terminal" refers to a function that uploads the generated video and audio content to cloud storage and provides a download link or streaming URL to the user terminal.

[0895] The "means for removing unnecessary information" has a function of performing preprocessing to remove unimportant information and special characters that become noise from the analyzed text data.

[0896] "Means for converting to an audio file using a text-to-speech engine" means a means for converting the summarized text into audio using text-to-speech technology and generating an audio file.

[0897] The "means of converting the text into an animated video using video generation software" has the function of generating an animated video based on the summarized text, along with slides and visuals.

[0898] The "means for providing the contents of the factory robot's instruction manual in real time" has the function of presenting information on the handling and maintenance of the factory robot to workers in real time through smart glasses.

[0899] MODE FOR CARRYING OUT THE INVENTION

[0900] The present invention relates to a system for providing instruction manuals in a format that is easy for users to understand. This system analyzes the text of the instruction manual, converts the summarized content into video and audio formats, and provides the content in real time via smart glasses. An embodiment of this system will be specifically described below.

[0901] Program Overview

[0902] The system has the following main functions:

[0903] 1. Ability to analyze instruction manual text

[0904] 2. Ability to summarize text using generative AI models

[0905] 3. Ability to convert summarized text into video and audio formats

[0906] 4. Function to provide content to smart glasses and user devices

[0907] Hardware and Software Configuration

[0908] The hardware and software used includes:

[0909] Terminal: The device to which the user uploads the instruction manual, such as a PC or smartphone.

[0910] Server: A central server that performs text analysis, summarization, and content conversion. It uses Python, a PDF analysis library, GPT-4, a Text-to-Speech API, and a video generation API as its main software.

[0911] Smart glasses: devices that provide real-time visual and audio information, such as Google Glass.

[0912] Data processing and calculation

[0913] 1. Upload and analyze the instruction manual:

[0914] Users upload a PDF of the instruction manual through their device, and the server extracts the text from the PDF and removes unnecessary information.

[0915] 2. Summary generation:

[0916] The extracted text is summarized using a generative AI model (GPT-4). As a concrete example, the following prompt sentence is sent to the AI ​​model to generate a summary:

[0917] Please summarize the contents of the following instruction manual:

[0918] "This robot requires daily inspection. The inspection procedure is as follows: remove part A, clean part B, and reinstall part C."

[0919] 3. Content Transformation:

[0920] The summarized text is converted into audio using a Text-to-Speech API, and then an animated video is generated using a Video Generation API.

[0921] 4. Provision to smart glasses and user devices:

[0922] The generated audio and video content is uploaded to cloud storage and provided to users in real time via smart glasses, and a download link or streaming URL is provided to the user's device as needed.

[0923] Specific examples

[0924] For example, when a factory worker wearing smart glasses performs maintenance on a robot, he or she uploads a PDF instruction manual to the system. The server extracts text from the PDF and generates a summary using GPT-4. The summary is then converted into a video with audio and animation using the Text-to-Speech API and the Video Generation API. This audio and video is then provided to the worker in real time via the smart glasses. This allows the worker to visually confirm the procedure while receiving audio instructions.

[0925] This system will enable factory workers to significantly reduce the time it takes to understand complex instruction manuals, enabling them to work efficiently and safely.

[0926] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0927] Step 1:

[0928] A user accesses the web or mobile application. The user clicks the "Upload" button, selects the instruction manual file (PDF or text file), and uploads it.

[0929] Input: Instruction manual file

[0930] Output: Binary data of the instruction manual

[0931] Specific operation: The user selects the desired instruction manual from the file selection dialog and presses the upload button. This action sends the binary data of the instruction manual to the terminal.

[0932] Step 2:

[0933] The terminal transmits the binary data of the instruction manual selected by the user to the server.

[0934] Input: Binary data of instruction manual

[0935] Output: Data sent to the server

[0936] Specific operation: The device sends the binary data of the instruction manual to the server as an HTTP POST request. This request uploads the file to the server.

[0937] Step 3:

[0938] The server analyzes the received instruction manual file and extracts the text data. If it is a PDF file, it uses a PDF analysis library to extract the text, and if it is a text file, it reads it directly.

[0939] Input: Binary data of the instruction manual uploaded to the server

[0940] Output: Extracted text data

[0941] What it does: The server uses a PDF parsing library to extract text from PDF files, or if it's a text file, reads the text data directly.

[0942] Step 4:

[0943] The server preprocesses the extracted text data to remove unnecessary information and special characters.

[0944] Input: Extracted text data

[0945] Output: Preprocessed clean text data

[0946] Specific operation: The server cleanses the text data using regular expressions and string manipulation to remove unnecessary information and special characters.

[0947] Step 5:

[0948] The server inputs the preprocessed text data into a generative AI model (GPT-4) to generate a summary.

[0949] Input: Preprocessed clean text data

[0950] Output: Summary text

[0951] Specific operation: The server inputs the following prompt to the generated AI model:

[0952] Please summarize the contents of the following instruction manual:

[0953] "This is an instruction manual divided into sections for setup, usage, and troubleshooting. The setup is as follows..."

[0954] A generative AI model generates a summary based on this prompt.

[0955] Step 6:

[0956] The server converts the generated summary text into audio using a Text-to-Speech API.

[0957] Input: Summary text

[0958] Output: Audio file (mp3, etc.)

[0959] Specific operation: The server sends the summary text to the Text-to-Speech API and receives the generated audio file.

[0960] Step 7:

[0961] The server generates animated videos based on the summary text using a video generation API.

[0962] Input: Summary text

[0963] Output: Animated video file

[0964] Specific operation: The server sends the summary text to the video generation API and receives the generated video file.

[0965] Step 8:

[0966] The server uploads the generated audio and video files to cloud storage and generates a link to provide to the user terminal and smart glasses.

[0967] Input: Audio and video files

[0968] Output: Upload and download links to cloud storage, streaming URL

[0969] Specific operation: The server uploads the audio and video files to a cloud storage service and provides the generated link to the user's application and smart glasses app.

[0970] Step 9:

[0971] Users can view and listen to the contents of the instruction manual summarized in real time through smart glasses and devices.

[0972] Input: Download link / Streaming URL

[0973] Output: Real-time viewing and listening content

[0974] What it does: The user clicks on the provided link or accesses the viewing screen with smart glasses to view audio and video content in real time.

[0975] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0976] The present invention is a system that not only summarizes instruction manuals using a generative AI model and converts the summaries into video and audio formats, but also combines it with an emotion engine that recognizes user emotions to provide more user-friendly content. Specific embodiments of this system are described below.

[0977] User Operation

[0978] First, users access the system and upload the instruction manual file (PDF or text file). Users click the "Upload" button in their web browser or mobile application and select the instruction manual they want to upload from the file selection dialog.

[0979] Terminal operation

[0980] The device reads the instruction manual file selected by the user, converts it into binary data, and then sends the file data to the server as an HTTP POST request to request analysis.

[0981] Server Processing

[0982] The server receives the file sent from the device and parses its contents. This involves extracting text from the PDF file using a PDF parsing library, or reading the text directly if it is a text file. The server then preprocesses the plain text, removing unnecessary whitespace and special characters.

[0983] The parsed text is then summarized using a generative AI model (e.g., GPT-4) to extract key points from the instruction manual and generate a concise summary. The final summary is evaluated and regenerated if necessary.

[0984] Content Conversion and Delivery

[0985] The summarized text is then converted into video or audio format. The server first inputs the summarized text into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. Then, video generation software is used to generate an animated video based on the summarized content. This video includes text slides and visuals, and also integrates the generated audio file.

[0986] Incorporating an emotion engine

[0987] The emotion engine recognizes the user's emotional state in real time and adjusts the content of the summary text and the tone and speed of the generated video and audio based on its output. For example, if the user feels confused or anxious about the information, the server will provide more detailed summary content at a slower pace based on the output of the emotion engine. The data from the emotion engine will also be used to adjust the animation and audio tone to reduce the user's psychological burden.

[0988] Content Submissions and Feedback

[0989] Finally, the server uploads the generated video and audio files to cloud storage and generates and sends a download link or streaming URL to the user's device. The user uses this link to watch the video or audio. The emotion engine monitors the user's reactions in real time while watching and adjusts the content as needed.

[0990] Specific examples

[0991] For example, consider a case where a user uploads a user manual for a home appliance (e.g., a smartphone). The user accesses a web application, clicks the "Upload" button, and selects the PDF file of the smartphone's user manual. The device then sends this file to a server, which extracts text from the PDF and uses a generative AI model to generate a summary of key sections.

[0992] The server then inputs the summarized text into a text-to-speech engine to generate an audio file, and uses video generation software to create an animated video. During this process, an emotion engine recognizes the user's emotions and adjusts the summary and video tone and speed as needed. For example, if the user is confused, the explanation will be more detailed and the video will be slower.

[0993] Finally, the server uploads the generated video and audio files to cloud storage and provides a download link to the user. The user clicks on this link to watch the video and understand how to set up the smartphone and use its functions. In this way, the present invention is a system that converts complex instruction manuals into a user-friendly format and provides them adaptively using an emotion engine.

[0994] The processing flow will be explained below.

[0995] Step 1:

[0996] The user starts uploading the instruction manual.

[0997] Users click the "Upload" button in their web browser or mobile application and select the instruction manual (PDF or text file) they want to upload from the file selection dialog.

[0998] Step 2:

[0999] The device sends the uploaded file to the server.

[1000] The terminal reads the selected file, converts it to binary data, then generates an HTTP POST request and sends it to the server with the file data.

[1001] Step 3:

[1002] The server receives and parses the file.

[1003] The server receives the HTTP request and extracts the attachment data. For PDF files, the server extracts the text using a PDF parsing library. For text files, the server reads the text data directly. The extracted text data is then preprocessed to remove unnecessary whitespace and special characters.

[1004] Step 4:

[1005] The server generates the summary using the generative AI model.

[1006] The server loads the generative AI model and feeds it preprocessed text data. The AI ​​model extracts key information from this data and generates a concise summary. The generated summary is evaluated and regenerated if necessary.

[1007] Step 5:

[1008] The server uses an emotion engine to recognize the user's emotional state.

[1009] The server uses an emotion engine to collect and analyze the user's real-time emotion data, such as facial expressions, tone of voice, and eye movements, to understand the user's emotional state.

[1010] Step 6:

[1011] The server converts the summary into video or audio format.

[1012] The server inputs the summarized text into a text-to-speech engine to generate an audio file, and then uses video generation software to generate an animated video based on the summarized content, adjusting the tone and speed of the video and audio accordingly based on data from the emotion engine.

[1013] Step 7:

[1014] The server provides the generated video and audio files to the user.

[1015] The server uploads the generated video and audio files to cloud storage and generates and sends a download link or streaming URL to the user's device.

[1016] Step 8:

[1017] View and listen to user-generated content.

[1018] Users can download or stream the video and audio using the provided link, and can pause, rewind, or fast-forward to the section they want while watching.

[1019] Step 9:

[1020] The server continuously monitors the user's emotional state.

[1021] The server monitors the user's emotional state in real time while watching video or listening to audio, collecting feedback and adjusting or regenerating content as needed based on this data.

[1022] This makes it possible to realize a system that provides the contents of the instruction manual to the user in an optimal format and makes it easy to understand.

[1023] Example 2

[1024] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1025] Conventional instruction manuals are often complex and difficult to understand, making it difficult for users to quickly obtain the information they need. Furthermore, conventional technologies provide information in a uniform format without considering the user's emotional state, making it difficult to provide a user-friendly experience. Furthermore, technologies for converting instruction manual summaries into video or audio formats are limited, making it difficult to meet the diverse needs of users.

[1026] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for analyzing the instruction manual document, means for summarizing the analyzed document using a generative AI model, means for converting the summarized document into video and audio format, means for uploading the converted content to cloud storage to be provided to the user terminal, and means for recognizing the user's emotional state using an emotion engine and adjusting the summary content and the tone and speed of the content. This not only enables the user to quickly obtain the information they need, but also provides the information in an appropriate format and at an appropriate pace depending on the user's emotional state, resulting in a more user-friendly experience.

[1027] An "instruction manual" is a document provided to explain how to use or operate a product or service.

[1028] A "document" is a collection of text or information written to convey a specific message.

[1029] "Analysis" is the process of examining data or information in detail to understand its structure and meaning.

[1030] A "generative AI model" is a model that uses artificial intelligence technology to generate new text or data.

[1031] A "summary" is a concise summary of the important parts of a document or data.

[1032] A "video format" is a data format that expresses movement by playing back a series of images or videos over time.

[1033] An "audio format" is a format in which sound or voice is represented as digital data.

[1034] "Cloud storage" is a remote data storage service that allows you to store and access data over the Internet.

[1035] An "emotion engine" is a technology / system that analyzes a user's emotional state in real time and utilizes the results.

[1036] A "user terminal" is hardware or a device that a user directly operates to input or receive information.

[1037] "Tone" refers to the tone, timbre, and atmosphere of a voice or text.

[1038] "Speed" refers to the speed at which information or content is provided to users.

[1039] The present invention is a system for analyzing an instruction manual, summarizing the contents thereof, and providing the summarization to a user. A specific embodiment of this system will be described below.

[1040] System Overview

[1041] Hardware Configuration

[1042] This system consists of three main components: user terminals, servers, and cloud storage. User terminals are devices such as PCs, smartphones, and tablets, while servers are computing devices for analysis and generation processing. Cloud storage is a remote data storage service that stores generated content and provides it to users.

[1043] Software Configuration

[1044] The system includes a PDF parsing library, a generative AI model (e.g., GPT-4), a text-to-speech engine (e.g., Google Text-to-Speech), video generation software, and an emotion engine.

[1045] Processing flow

[1046] User Operation

[1047] Users use a web browser or mobile application to upload instruction manual files (PDF or text files) to the system. Specifically, they access the system, click the "Upload" button, and select the instruction manual they want to upload from the file selection dialog.

[1048] Terminal operation

[1049] The device reads the instruction manual file selected by the user, converts it into binary data, and then sends the file data to the server as an HTTP POST request.

[1050] Server Processing

[1051] The server receives the file sent from the device and analyzes its contents. For PDF files, it extracts the text using a PDF analysis library, while for text files, it reads the text data directly. It then summarizes the analyzed text using a generative AI model. The generative AI model used (e.g., GPT-4) is given a prompt such as "Please briefly summarize the key points of this instruction manual" to generate an appropriate summary.

[1052] Content generation

[1053] The server inputs the summarized text into a text-to-speech engine to generate an audio file, and then uses video generation software to generate an animated video based on the summarized content, incorporating text slides and visuals and the generated audio file.

[1054] Emotion Engine Operation

[1055] The emotion engine recognizes the user's emotional state in real time. For example, if the user is using a webcam, it uses facial expression recognition technology to analyze their emotions and sends the results to the server. Based on the results, the server adjusts the content of the summary text and the tone and speed of the generated content.

[1056] Content provision

[1057] The server uploads the generated video and audio files to cloud storage and provides a download link or streaming URL to the user's device. The user uses this link to watch the video or audio. During viewing, the emotion engine monitors the user's reactions in real time and adjusts the content as needed.

[1058] Specific examples

[1059] For example, consider a user uploading a user manual for a home appliance (e.g., a smartphone). The user accesses a web application, clicks the "Upload" button, and selects the PDF file of the smartphone's user manual. The device sends this file to a server, which extracts text from the PDF and uses a generative AI model to generate a summary of key sections. The server then inputs the summarized text into a text-to-speech engine to generate an audio file. It then uses video generation software to create an animated video. During this process, an emotion engine recognizes the user's emotions and adjusts the summary content, tone, and speed of the video as needed. Finally, the server uploads the generated video and audio files to cloud storage and provides the user with a download link. The user clicks the link to watch the video and learn how to set up the smartphone and use its features.

[1060] Prompt Sentence Examples

[1061] "Please briefly summarize the important points of this instruction manual."

[1062] The above is a detailed description of the embodiment of the present invention. This system can convert complex instruction manuals into a user-friendly format and provide them adaptively using an emotion engine.

[1063] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1064] Step 1:

[1065] The user uploads an instruction manual to the system. The user accesses the system, clicks the "Upload" button using a web browser or mobile application, and selects the instruction manual file (PDF or text file) from the file selection dialog. The input is the instruction manual file, and the output is the path to the selected file.

[1066] Step 2:

[1067] The terminal reads the selected file and converts it to binary data. The terminal uses the file system API to read the specified file and converts it to binary data. It then sends this binary data to the server as an HTTP POST request. The input is the path of the file selected by the user, and the output is the binary data sent to the server.

[1068] Step 3:

[1069] The server receives the binary data sent from the terminal, saves the file, and analyzes the contents. The server first saves the file in a temporary directory. Next, if it is a PDF file, it uses a PDF parsing library to extract the text, or if it is a text file, it reads the text data directly. The input is binary data, and the output is the extracted plain text. Specific operations include calling the API of the PDF parsing library to extract the text from the file.

[1070] Step 4:

[1071] The server summarizes the extracted text using a generative AI model. The server preprocesses the extracted plain text to remove unnecessary white space and special characters. The generative AI model is then given a prompt sentence, "Please briefly summarize the important points of this instruction manual," to generate summarized text. The input is the parsed plain text, and the output is the summarized text.

[1072] Step 5:

[1073] The server converts the summarized text into audio format. The server inputs the summarized text into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. The input is the summarized text, and the output is an audio file. Specifically, the Google Text-to-Speech API is used to convert text into audio.

[1074] Step 6:

[1075] The server generates a video that includes an audio file. The server uses video generation software to create an animated video by integrating text slides and visuals with the generated audio file. The input is the summarized text and audio file, and the output is the generated video file. Specifically, the video generation software runs a script to generate the video.

[1076] Step 7:

[1077] The emotion engine recognizes the user's emotional state in real time. If the user is using a webcam, the emotion engine uses facial expression recognition technology to analyze emotions and sends the results to the server. The input is the video feed from the webcam, and the output is the analyzed emotion data. Specifically, it runs a facial expression recognition algorithm to analyze the user's emotions.

[1078] Step 8:

[1079] The server adjusts the content based on the output of the emotion engine. The server uses the emotion data to adjust the content of the summary text and the tone and speed of the generated video and audio. The input is emotion data, and the output is the adjusted content. Specifically, the server changes the parameters of the generation algorithm in real time.

[1080] Step 9:

[1081] The server uploads the generated video and audio files to cloud storage. The server uploads the files using a cloud storage service and provides an access link to the user's device. The input is the generated content file, and the output is the upload result to cloud storage and the access link. Specifically, the server uses the cloud storage API to upload the files and generate a link.

[1082] Step 10:

[1083] The user views the content using the provided link. The user clicks on a download link or streaming URL to view the video or audio. The input is the access link, and the output is the content to be viewed. The specific operation is to play the content using a browser or media player.

[1084] The above is a detailed flow of the program processing for this system. The input and output at each step are clearly stated, and the type of data processing and calculation that is performed is specifically explained.

[1085] (Application example 2)

[1086] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1087] In recent years, many factories have introduced new robots, but complicated instruction manuals hinder workers' quick understanding, resulting in a decline in work efficiency. Furthermore, there is a lack of technology to adapt instructions according to the user's emotional state, which often leaves workers confused.

[1088] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing the text of the instruction manual, means for summarizing the analyzed text using a generative AI model, means for converting the summarized text into video and audio format, means for providing the converted content to a user terminal, means for recognizing the user's emotional state in real time, and means for adjusting the content of the summarized text and the tone and speed of the video and audio based on the user's emotional state. This makes it possible to improve work efficiency and reduce user confusion.

[1089] An "instruction manual" is a document that contains information on how to use, maintain, and handle a product.

[1090] "Analyzing" means breaking down given data and examining each element in detail.

[1091] A "generative AI model" is a model that uses artificial intelligence techniques to generate new data, such as summarizing or automatically generating text.

[1092] To "summarize" means to summarize a longer piece of text or information in a concise form.

[1093] "Video format" is a format that expresses information as moving images.

[1094] "Audio format" is a format in which information is conveyed auditorily.

[1095] A "user terminal" is a device such as a computer or smartphone that is directly operated by a user.

[1096] "Emotional state" refers to a user's current psychological state or mood.

[1097] "Real-time" is a term that refers to processing or response occurring immediately, as in actual time.

[1098] "Tone" is an expression that refers to the tone or color of a sound or voice.

[1099] "Speed" refers to the speed at which information is transmitted within a given period of time.

[1100] This system summarizes the complex content of instruction manuals, converts them into video and audio formats, and adaptively adjusts the content according to the user's emotions, allowing factory workers to easily understand the instruction manuals for newly installed robots and improving work efficiency.

[1101] System configuration

[1102] 1. User Operation

[1103] First, the user uploads the instruction manual (PDF or text file) using the smartphone app. Click the "Upload" button in the app and select the file to upload from the file selection dialog.

[1104] 2. Terminal operation

[1105] The uploaded file is read by the smartphone and converted into binary data, which is then sent to the server as an HTTP POST request.

[1106] 3. Server Processing

[1107] The server uses a PDF parsing library (e.g., fitz) to extract text from the PDF file, or reads it directly in the case of text files.

[1108] Then, a generative AI model (e.g., GPT-4) is used to extract key points from the plain text and generate a concise summary. A text-to-speech engine (e.g., gTTS) is then used to generate an audio file, which is then used to create an animated video using video generation software (e.g., moviepy).

[1109] 4. Emotion engine integration

[1110] The emotion engine recognizes the user's emotional state in real time through the user's camera feed. For example, if the user is confused, the server will adjust the summary, tone and speed of the video to provide a more detailed and slower explanation.

[1111] 5. Content Submissions and Feedback

[1112] The generated video and audio files are uploaded to cloud storage, and a download link or streaming URL is generated for the user. The user can use the link to watch the video or audio. The emotion engine monitors the user's reactions in real time while watching and adjusts the content as needed.

[1113] Specific examples

[1114] For example, a factory worker uploads a PDF instruction manual for a new robot to the app. The server extracts the text from the PDF and uses a generative AI model to summarize the key points of the manual. The summarized text is then fed into a text-to-speech engine to generate an audio file, which is then used by video generation software to create an animated video.

[1115] The emotion engine monitors the user's emotional state in real time and regenerates the summary content if they are confused, providing more detailed and slower explanations.

[1116] Prompt Sentence Examples

[1117] Summarize the following text:

[1118] [Text extracted from PDF]

[1119] It should be around 100-200 characters in length, focus on the key points, and be concise.

[1120] As described above, this system contributes to improving work efficiency by converting complex instruction manuals into an easily understandable format and providing adaptive explanations using an emotion engine.

[1121] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1122] Step 1:

[1123] The user uploads an instruction manual (PDF or text file) using a smartphone app. The input data is the instruction manual file on the user's device, and the output data is binary data sent to the server.

[1124] Step 2:

[1125] The terminal reads the uploaded file, converts it to binary data, and sends the converted binary data to the server as an HTTP POST request. The input is the file selected by the user, and the output is the binary data sent to the server.

[1126] Step 3:

[1127] The server parses the received binary data and extracts the text from the PDF file using a PDF parsing library (e.g., fitz) or reads the data directly from the text file. The input is the binary data sent to the server, and the output is the extracted text data.

[1128] Step 4:

[1129] The server generates a summary from the plain text using a generative AI model (e.g., GPT-4). In this process, the input is the extracted text data, and the output is the generated summary text. Specifically, a prompt sentence is input to the generative AI model, and the model performs a process of summarizing the key points.

[1130] Step 5:

[1131] The server inputs the generated summary text into a text-to-speech engine (e.g., gTTS) to generate an audio file. The input is the summarized text data, and the output is an audio file.

[1132] Step 6:

[1133] The server creates an animated video using video generation software (e.g., moviepy) based on the generated audio file. The input is the audio file and summary text, and the output is the generated animated video file.

[1134] Step 7:

[1135] The emotion engine recognizes the user's emotional state in real time through the user's camera feed. The input is data from the user's camera feed, and the output is data on the user's emotional state. Based on the emotional state, it adjusts the tone and speed of the video and audio.

[1136] Step 8:

[1137] The server realigns the summary text and the generated video and audio files according to the output data of the emotion engine. The input is the emotion engine data, and the output is the realigned content.

[1138] Step 9:

[1139] Finally, the generated video and audio files are uploaded to cloud storage, and a download link or streaming URL is provided to the user's device. The input is the generated video and audio files, and the output is uploading them to cloud storage and providing a link to the user.

[1140] A concrete example would be a factory floor worker uploading a new robot instruction manual to an app, watching abridged video and audio, and receiving detailed, slow-fire explanations when the emotion engine detects confusion.

[1141] Example prompt sentence:

[1142] Summarize the following text:

[1143] [Text extracted from PDF]

[1144] It should be around 100-200 characters in length, focus on the key points, and be concise.

[1145] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1146] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1147] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1148] [Fourth embodiment]

[1149] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1150] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1151] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1152] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1153] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1154] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1155] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1156] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1157] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1158] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1159] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1160] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1161] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1162] The present invention relates to a system that uses a generative AI model to summarize instruction manuals and convert them into video and audio formats. The purpose of this system is to provide users with instruction manuals in a format that is easy to understand. Specific embodiments of this system are described below.

[1163] User Operation

[1164] Users access the system to upload instruction manual files (PDF or text files). Users click the "Upload" button in their web browser or mobile application and select the instruction manual they want to upload from the file selection dialog.

[1165] Terminal operation

[1166] The device reads the instruction manual file selected by the user and sends the binary data to the server. Specifically, it sends the file to the server as an HTTP POST request, requesting that the server analyze the instruction manual.

[1167] Server Processing

[1168] The server receives the file sent from the terminal and analyzes its contents. For analysis, an appropriate library (e.g., a PDF analysis library or text analysis algorithm) is used. For PDF files, text extraction is performed, and for text files, the text data is read directly. Next, the plain text is preprocessed to remove unnecessary information and special characters.

[1169] Next, we use a generative AI model to summarize the parsed text. We use a generative AI model (e.g., GPT-4) to extract key points from the instruction manual and generate a concise summary. We evaluate the summary results and regenerate them if necessary.

[1170] Content Conversion and Delivery

[1171] The summarized text is then converted into video and audio formats. The server first inputs the summarized text into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. Then, video generation software is used to generate an animated video, which includes slides and visuals based on the summarized content. The generated audio file is also incorporated into the video.

[1172] Finally, the generated video and audio files are uploaded to cloud storage, and a download link or streaming URL is provided to the user's device, allowing the user to view the video and audio and understand the contents of the instruction manual.

[1173] Specific examples

[1174] For example, consider the case where a user wants to upload an instruction manual for a home appliance (e.g., a washing machine) to the system. The user accesses a web application and clicks the "Upload" button. The user selects a PDF file of the washing machine's instruction manual from the local storage and uploads it. The device then sends this PDF file to the server, which extracts text from the PDF and performs preprocessing.

[1175] The server uses a generative AI model to extract only the key information and generate a summary divided into sections such as "Initial Setup," "How to Use," and "Troubleshooting." Based on this summary, a text-to-speech engine is used to generate an audio file, and video generation software is used to create an animated video.

[1176] Finally, the server uploads the generated video and audio files to cloud storage and provides a download link to the user. The user can then use the provided link to watch the video and understand how to set up and use the washing machine. In this way, a system is realized that converts complex instruction manuals into a user-friendly format, facilitating user understanding.

[1177] The processing flow will be explained below.

[1178] Step 1:

[1179] The user starts uploading the instruction manual.

[1180] Users click the "Upload" button in their web browser or mobile application and select the instruction manual (PDF or text file) they want to upload from the file selection dialog.

[1181] Step 2:

[1182] The device sends the uploaded file to the server.

[1183] The terminal reads the selected file, converts it to binary data, and then generates an HTTP POST request and sends it to the server with the file data.

[1184] Step 3:

[1185] The server receives and parses the file.

[1186] The server receives the HTTP request and extracts the attached file data. For PDF files, the server extracts the text using a PDF parsing library. For text files, the server reads the text data directly. The extracted text data is then preprocessed to remove unnecessary whitespace and special characters.

[1187] Step 4:

[1188] The server generates the summary using the generative AI model.

[1189] The server loads the generative AI model and feeds it preprocessed text data. The AI ​​model extracts key information from this data and generates a concise summary. The summary is evaluated and regenerated if necessary.

[1190] Step 5:

[1191] The server converts the summary into video or audio format.

[1192] The server inputs the summarized text into a text-to-speech engine to generate an audio file, then uses video generation software to generate an animated video based on the summarized content, adding text slides and visuals to the video and incorporating the generated audio file.

[1193] Step 6:

[1194] The server provides the generated video and audio files to the user.

[1195] The server uploads the generated video and audio files to cloud storage and generates and sends a download link or streaming URL to the user's device.

[1196] Step 7:

[1197] View and listen to user-generated content.

[1198] Users can download or stream the video and audio using the provided link, and can pause, rewind, or fast-forward to the section they want while watching.

[1199] Through these steps, a system is realized that converts the contents of an instruction manual into a concise, easy-to-understand video or audio format and provides it to users.

[1200] Example 1

[1201] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1202] The instruction manuals that accompany a wide variety of modern products are often complex and require time and effort for users to understand. In particular, if they contain long texts or a lot of technical terminology, it can be difficult for users to understand, and they may not use or maintain the product properly. Therefore, there is a need for a method to communicate the contents of the instruction manual efficiently and effectively.

[1203] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1204] In this invention, the server includes a means for analyzing data of the instruction manual, a means for summarizing the analyzed data using a generative AI model, and a means for converting the summarized data into audio and video formats, thereby enabling the user to understand the contents of the instruction manual concisely and efficiently.

[1205] An "instruction manual" is a document that provides information on how to use a product or service, installation procedures, maintenance methods, troubleshooting, etc.

[1206] "Data" means content expressed in the form of information, instructions, numbers, etc., that is stored and transmitted electronically.

[1207] "Analysis" is the process of breaking down complex data and information and converting it into an understandable form.

[1208] A "generative AI model" is an artificial intelligence algorithm that uses deep learning and natural language processing to generate new data and text.

[1209] A "summary" is a concise summary of a large amount of information, extracting only the main points and shortening them.

[1210] "Speech" is the electronic representation of spoken words or other acoustic signals.

[1211] "Video" refers to the electronic representation of visual information, including text, images, animations, etc.

[1212] "User" means a person or entity that uses a system or product.

[1213] A "terminal" is an electronic device that allows a user to access and operate the system.

[1214] "Offering" is the act of giving something to someone else or making it available for their use.

[1215] The present invention relates to a system that uses a generative AI model to summarize instruction manuals and convert them into audio and video formats. The purpose of this system is to provide users with instruction manuals in a format that is easy to understand. Specific embodiments of this system are described below.

[1216] User Operation

[1217] Users access the system to upload instruction manual files (PDF or text files). They click the "Upload" button in their web browser or mobile application and select the instruction manual they want to upload from the file selection dialog. Once the upload is complete, the device prepares to send the file to the server.

[1218] Terminal operation

[1219] The device loads the instruction manual file selected by the user and sends the binary data to the server. Specifically, the device creates an HTTP POST request to send the selected file to the server as binary data. This request also includes metadata such as the file name and file type.

[1220] Server Processing

[1221] The server receives the file sent from the terminal. After receiving it, it identifies the file type and selects the appropriate parsing method. If it is a PDF file, it uses a PDF parsing library (e.g., PDFBox or PyPDF2) to extract the text data. If it is a text file, it reads the text data directly.

[1222] The server then preprocesses the extracted text data, removing unnecessary information (e.g., advertisements or specific product brands) and special characters, and checking the accuracy of grammar and syntax, correcting errors if necessary.

[1223] The server inputs the preprocessed text into a generative AI model (e.g., OpenAI's GPT-4) to generate a summary. Specifically, it invokes the generative AI model and gives it the following prompt:

[1224] Prompt: "Summarize the text from the instruction manual below. Extract the key points and summarize them concisely."

[1225] The generative AI model extracts important information based on the input text and generates a summary.

[1226] Content Conversion and Delivery

[1227] The server then converts the generated summary into audio and video formats. First, the summarized text is fed into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. Then, video generation software (e.g., Adobe After Effects or Apple Motion) is used to create an animated video based on the audio data. Slides and visuals based on the summarized content are added to the video.

[1228] Finally, the server uploads the generated video and audio files to cloud storage (e.g., Amazon S3 or Google Cloud Storage), obtains the file URL on the cloud storage, and provides a download link or streaming URL to the user's device.

[1229] Users can use the links to view videos and audio, which allows them to easily understand the contents of the instruction manual through both sight and sound.

[1230] Specific examples

[1231] For example, consider the case where a user wants to upload an instruction manual for a home appliance (e.g., a washing machine). The user accesses a web application, clicks the "Upload" button, selects the PDF file of the washing machine's instruction manual from the local storage, and uploads it. The device then sends this PDF file to the server, which extracts text from the PDF and performs preprocessing.

[1232] The server uses a generative AI model to extract only the key information and generate a summary divided into sections such as "Initial Setup," "How to Use," and "Troubleshooting." Based on this summary, a text-to-speech engine is used to generate an audio file, and video generation software is used to create an animated video.

[1233] Finally, the server uploads the generated video and audio files to cloud storage and provides a download link to the user. The user can then use the provided link to watch the video and understand how to set up and use the washing machine. This system makes it possible to provide complex instruction manuals in a concise, visually easy-to-understand format.

[1234] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1235] Step 1: User interaction

[1236] Users access the system and upload instruction manual files (PDF or text files). Specifically, they click the "Upload" button in their web browser or mobile application and select the instruction manual they want to upload from the file selection dialog. The input is the instruction manual file specified by the user, and the output is the binary data of the selected file.

[1237] Step 2: Loading and sending the file

[1238] The terminal reads the instruction manual file selected by the user and sends the binary data to the server. Specifically, the terminal creates an HTTP POST request and sends the selected file as binary data to the server. The input is the binary data of the file selected by the user, and the output is the HTTP POST request sent to the server.

[1239] Step 3: Receiving and analyzing the file

[1240] The server receives the file sent from the terminal. After receiving it, it identifies the file type and selects the appropriate parsing method. If it is a PDF file, it uses a PDF parsing library (e.g., PDFBox or PyPDF2) to extract the text data. If it is a text file, it reads the text data directly. The input is the file data received by the server, and the output is the extracted text data.

[1241] Step 4: Preprocessing the data

[1242] The server preprocesses the extracted text data. This preprocessing removes unnecessary information (e.g., advertisements or specific product brands) and special characters. It also checks the accuracy of grammar and syntax and corrects errors as needed. The input is the extracted text data, and the output is the preprocessed, clean text data.

[1243] Step 5: Summarizing the text

[1244] The server inputs the preprocessed text into a generative AI model (e.g., GPT-4) to generate a summary. Specifically, it invokes the generative AI model and gives it the following prompt:

[1245] Prompt: "Summarize the text from the instruction manual below. Extract the key points and summarize them concisely."

[1246] The input is preprocessed text data and a prompt sentence, and the output is the generated summary sentence.

[1247] Step 6: Converting the summary into audio and video formats

[1248] The server inputs the generated summary into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. It then uses video generation software (e.g., Adobe After Effects or Apple Motion) to create an animated video, adding slides and visuals based on the summarized content. The input is the summary, and the output is the generated audio and video files.

[1249] Step 7: Upload and provide files

[1250] The server uploads the generated video and audio files to cloud storage (e.g., Amazon S3 or Google Cloud Storage). It obtains the file URL on the cloud storage and provides a download link or streaming URL to the user's device. The input is the generated audio and video files, and the output is the download link or streaming URL provided to the user.

[1251] Specific examples

[1252] For example, if a user wants to upload a washing machine instruction manual, the actual operation steps are as follows:

[1253] Step 1: The user accesses the web application, clicks the "Upload" button, selects the washing machine instruction manual (PDF file), and clicks the "Submit" button.

[1254] Step 2: The device reads the selected PDF file and creates an HTTP POST request to send to the server.

[1255] Step 3: The server receives the file and uses PDFBox to extract and preprocess the text data.

[1256] Step 4: The server inputs the preprocessed text into the generative AI model to generate a summary.

[1257] Step 5: The server feeds the summary into Google Text-to-Speech to generate an audio file, and then uses Adobe After Effects to generate a video of the slides and audio synchronized to the text.

[1258] Step 6: The server uploads the generated video and audio files to Amazon S3 and provides a download link to the user.

[1259] Step 7: The user uses the provided link to watch and listen to videos and audio to understand how to set up and use the washing machine.

[1260] This series of processes provides the instruction manual in a concise and visually easy-to-understand format.

[1261] (Application example 1)

[1262] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1263] Conventional instruction manuals require a huge amount of time and effort for users to understand their contents, which is a major obstacle to efficient work, especially on factory floors. Furthermore, paper-based or simple text-based manuals make it difficult to visually understand procedures and operations, increasing the risk of operational errors and maintenance mistakes. This can lead to reduced production efficiency and safety, so a more user-friendly instruction manual provision system is needed.

[1264] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1265] In this invention, the server includes a means for analyzing the text of the instruction manual, a means for summarizing the analyzed text using a generative AI model, a means for converting the summarized text into video and audio formats, a means for providing the summarized content to a user via smart glasses in visual and audio formats, and a means for providing the converted content to a user terminal, thereby enabling the user to understand the instruction manual and maintenance procedures of a factory robot in real time through visual and audio formats, thereby achieving efficient and safe work.

[1266] The "means for analyzing the text of the instruction manual" has the function of reading the document data of the instruction manual uploaded by the user and analyzing its contents.

[1267] "Means for summarizing analyzed text using a generative AI model" refers to a function that has the function of concisely summarizing the contents of analyzed text data through a generative AI model that uses natural language processing technology.

[1268] A "means for converting summarized text into video and audio formats" is a means for using a text-to-speech engine or video generation software to create audio files and animated videos based on the summarized text.

[1269] The "means for visually and audibly providing summarized content to a user via smart glasses" means a means for providing summarized text content to the user's visual and auditory senses in real time via smart glasses.

[1270] The "means for providing converted content to a user terminal" refers to a function that uploads the generated video and audio content to cloud storage and provides a download link or streaming URL to the user terminal.

[1271] The "means for removing unnecessary information" has a function of performing preprocessing to remove unimportant information and special characters that become noise from the analyzed text data.

[1272] "Means for converting to an audio file using a text-to-speech engine" means a means for converting the summarized text into audio using text-to-speech technology and generating an audio file.

[1273] The "means of converting the text into an animated video using video generation software" has the function of generating an animated video based on the summarized text, along with slides and visuals.

[1274] The "means for providing the contents of the factory robot's instruction manual in real time" has the function of presenting information on the handling and maintenance of the factory robot to workers in real time through smart glasses.

[1275] MODE FOR CARRYING OUT THE INVENTION

[1276] The present invention relates to a system for providing instruction manuals in a format that is easy for users to understand. This system analyzes the text of the instruction manual, converts the summarized content into video and audio formats, and provides the content in real time via smart glasses. An embodiment of this system will be specifically described below.

[1277] Program Overview

[1278] The system has the following main functions:

[1279] 1. Ability to analyze instruction manual text

[1280] 2. Ability to summarize text using generative AI models

[1281] 3. Ability to convert summarized text into video and audio formats

[1282] 4. Function to provide content to smart glasses and user devices

[1283] Hardware and Software Configuration

[1284] The hardware and software used includes:

[1285] Terminal: The device to which the user uploads the instruction manual, such as a PC or smartphone.

[1286] Server: A central server that performs text analysis, summarization, and content conversion. It uses Python, a PDF analysis library, GPT-4, a Text-to-Speech API, and a video generation API as its main software.

[1287] Smart glasses: devices that provide real-time visual and audio information, such as Google Glass.

[1288] Data processing and calculation

[1289] 1. Upload and analyze the instruction manual:

[1290] Users upload a PDF of the instruction manual through their device, and the server extracts the text from the PDF and removes unnecessary information.

[1291] 2. Summary generation:

[1292] The extracted text is summarized using a generative AI model (GPT-4). As a concrete example, the following prompt sentence is sent to the AI ​​model to generate a summary:

[1293] Please summarize the contents of the following instruction manual:

[1294] "This robot requires daily inspection. The inspection procedure is as follows: remove part A, clean part B, and reinstall part C."

[1295] 3. Content Transformation:

[1296] The summarized text is converted into audio using a Text-to-Speech API, and then an animated video is generated using a Video Generation API.

[1297] 4. Provision to smart glasses and user devices:

[1298] The generated audio and video content is uploaded to cloud storage and provided to users in real time via smart glasses, and a download link or streaming URL is provided to the user's device as needed.

[1299] Specific examples

[1300] For example, when a factory worker wearing smart glasses performs maintenance on a robot, he or she uploads a PDF instruction manual to the system. The server extracts text from the PDF and generates a summary using GPT-4. The summary is then converted into a video with audio and animation using the Text-to-Speech API and the Video Generation API. This audio and video is then provided to the worker in real time via the smart glasses. This allows the worker to visually confirm the procedure while receiving audio instructions.

[1301] This system will enable factory workers to significantly reduce the time it takes to understand complex instruction manuals, enabling them to work efficiently and safely.

[1302] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1303] Step 1:

[1304] A user accesses the web or mobile application. The user clicks the "Upload" button, selects the instruction manual file (PDF or text file), and uploads it.

[1305] Input: Instruction manual file

[1306] Output: Binary data of the instruction manual

[1307] Specific operation: The user selects the desired instruction manual from the file selection dialog and presses the upload button. This action sends the binary data of the instruction manual to the terminal.

[1308] Step 2:

[1309] The terminal transmits the binary data of the instruction manual selected by the user to the server.

[1310] Input: Binary data of instruction manual

[1311] Output: Data sent to the server

[1312] Specific operation: The device sends the binary data of the instruction manual to the server as an HTTP POST request. This request uploads the file to the server.

[1313] Step 3:

[1314] The server analyzes the received instruction manual file and extracts the text data. If it is a PDF file, it uses a PDF analysis library to extract the text, and if it is a text file, it reads it directly.

[1315] Input: Binary data of the instruction manual uploaded to the server

[1316] Output: Extracted text data

[1317] What it does: The server uses a PDF parsing library to extract text from PDF files, or if it's a text file, reads the text data directly.

[1318] Step 4:

[1319] The server preprocesses the extracted text data to remove unnecessary information and special characters.

[1320] Input: Extracted text data

[1321] Output: Preprocessed clean text data

[1322] Specific operation: The server cleanses the text data using regular expressions and string manipulation to remove unnecessary information and special characters.

[1323] Step 5:

[1324] The server inputs the preprocessed text data into a generative AI model (GPT-4) to generate a summary.

[1325] Input: Preprocessed clean text data

[1326] Output: Summary text

[1327] Specific operation: The server inputs the following prompt to the generated AI model:

[1328] Please summarize the contents of the following instruction manual:

[1329] "This is an instruction manual divided into sections for setup, usage, and troubleshooting. The setup is as follows..."

[1330] A generative AI model generates a summary based on this prompt.

[1331] Step 6:

[1332] The server converts the generated summary text into audio using a Text-to-Speech API.

[1333] Input: Summary text

[1334] Output: Audio file (mp3, etc.)

[1335] Specific operation: The server sends the summary text to the Text-to-Speech API and receives the generated audio file.

[1336] Step 7:

[1337] The server generates animated videos based on the summary text using a video generation API.

[1338] Input: Summary text

[1339] Output: Animated video file

[1340] Specific operation: The server sends the summary text to the video generation API and receives the generated video file.

[1341] Step 8:

[1342] The server uploads the generated audio and video files to cloud storage and generates a link to provide to the user terminal and smart glasses.

[1343] Input: Audio and video files

[1344] Output: Upload and download links to cloud storage, streaming URL

[1345] Specific operation: The server uploads the audio and video files to a cloud storage service and provides the generated link to the user's application and smart glasses app.

[1346] Step 9:

[1347] Users can view and listen to the contents of the instruction manual summarized in real time through smart glasses and devices.

[1348] Input: Download link / Streaming URL

[1349] Output: Real-time viewing and listening content

[1350] What it does: The user clicks on the provided link or accesses the viewing screen with smart glasses to view audio and video content in real time.

[1351] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1352] The present invention is a system that not only summarizes instruction manuals using a generative AI model and converts the summaries into video and audio formats, but also combines it with an emotion engine that recognizes user emotions to provide more user-friendly content. Specific embodiments of this system are described below.

[1353] User Operation

[1354] First, users access the system and upload the instruction manual file (PDF or text file). Users click the "Upload" button in their web browser or mobile application and select the instruction manual they want to upload from the file selection dialog.

[1355] Terminal operation

[1356] The device reads the instruction manual file selected by the user, converts it into binary data, and then sends the file data to the server as an HTTP POST request to request analysis.

[1357] Server Processing

[1358] The server receives the file sent from the device and parses its contents. This involves extracting text from the PDF file using a PDF parsing library, or reading the text directly if it is a text file. The server then preprocesses the plain text, removing unnecessary whitespace and special characters.

[1359] The parsed text is then summarized using a generative AI model (e.g., GPT-4) to extract key points from the instruction manual and generate a concise summary. The final summary is evaluated and regenerated if necessary.

[1360] Content Conversion and Delivery

[1361] The summarized text is then converted into video or audio format. The server first inputs the summarized text into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. Then, video generation software is used to generate an animated video based on the summarized content. This video includes text slides and visuals, and also integrates the generated audio file.

[1362] Incorporating an emotion engine

[1363] The emotion engine recognizes the user's emotional state in real time and adjusts the content of the summary text and the tone and speed of the generated video and audio based on its output. For example, if the user feels confused or anxious about the information, the server will provide more detailed summary content at a slower pace based on the output of the emotion engine. The data from the emotion engine will also be used to adjust the animation and audio tone to reduce the user's psychological burden.

[1364] Content Submissions and Feedback

[1365] Finally, the server uploads the generated video and audio files to cloud storage and generates and sends a download link or streaming URL to the user's device. The user uses this link to watch the video or audio. The emotion engine monitors the user's reactions in real time while watching and adjusts the content as needed.

[1366] Specific examples

[1367] For example, consider a case where a user uploads a user manual for a home appliance (e.g., a smartphone). The user accesses a web application, clicks the "Upload" button, and selects the PDF file of the smartphone's user manual. The device then sends this file to a server, which extracts text from the PDF and uses a generative AI model to generate a summary of key sections.

[1368] The server then inputs the summarized text into a text-to-speech engine to generate an audio file, and uses video generation software to create an animated video. During this process, an emotion engine recognizes the user's emotions and adjusts the summary and video tone and speed as needed. For example, if the user is confused, the explanation will be more detailed and the video will be slower.

[1369] Finally, the server uploads the generated video and audio files to cloud storage and provides a download link to the user. The user clicks on this link to watch the video and understand how to set up the smartphone and use its functions. In this way, the present invention is a system that converts complex instruction manuals into a user-friendly format and provides them adaptively using an emotion engine.

[1370] The processing flow will be explained below.

[1371] Step 1:

[1372] The user starts uploading the instruction manual.

[1373] Users click the "Upload" button in their web browser or mobile application and select the instruction manual (PDF or text file) they want to upload from the file selection dialog.

[1374] Step 2:

[1375] The device sends the uploaded file to the server.

[1376] The terminal reads the selected file, converts it to binary data, then generates an HTTP POST request and sends it to the server with the file data.

[1377] Step 3:

[1378] The server receives and parses the file.

[1379] The server receives the HTTP request and extracts the attachment data. For PDF files, the server extracts the text using a PDF parsing library. For text files, the server reads the text data directly. The extracted text data is then preprocessed to remove unnecessary whitespace and special characters.

[1380] Step 4:

[1381] The server generates the summary using the generative AI model.

[1382] The server loads the generative AI model and feeds it preprocessed text data. The AI ​​model extracts key information from this data and generates a concise summary. The generated summary is evaluated and regenerated if necessary.

[1383] Step 5:

[1384] The server uses an emotion engine to recognize the user's emotional state.

[1385] The server uses an emotion engine to collect and analyze the user's real-time emotion data, such as facial expressions, tone of voice, and eye movements, to understand the user's emotional state.

[1386] Step 6:

[1387] The server converts the summary into video or audio format.

[1388] The server inputs the summarized text into a text-to-speech engine to generate an audio file, and then uses video generation software to generate an animated video based on the summarized content, adjusting the tone and speed of the video and audio accordingly based on data from the emotion engine.

[1389] Step 7:

[1390] The server provides the generated video and audio files to the user.

[1391] The server uploads the generated video and audio files to cloud storage and generates and sends a download link or streaming URL to the user's device.

[1392] Step 8:

[1393] View and listen to user-generated content.

[1394] Users can download or stream the video and audio using the provided link, and can pause, rewind, or fast-forward to the section they want while watching.

[1395] Step 9:

[1396] The server continuously monitors the user's emotional state.

[1397] The server monitors the user's emotional state in real time while watching video or listening to audio, collecting feedback and adjusting or regenerating content as needed based on this data.

[1398] This makes it possible to realize a system that provides the contents of the instruction manual to the user in an optimal format and makes it easy to understand.

[1399] Example 2

[1400] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1401] Conventional instruction manuals are often complex and difficult to understand, making it difficult for users to quickly obtain the information they need. Furthermore, conventional technologies provide information in a uniform format without considering the user's emotional state, making it difficult to provide a user-friendly experience. Furthermore, technologies for converting instruction manual summaries into video or audio formats are limited, making it difficult to meet the diverse needs of users.

[1402] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for analyzing the instruction manual document, means for summarizing the analyzed document using a generative AI model, means for converting the summarized document into video and audio format, means for uploading the converted content to cloud storage to be provided to the user terminal, and means for recognizing the user's emotional state using an emotion engine and adjusting the summary content and the tone and speed of the content. This not only enables the user to quickly obtain the information they need, but also provides the information in an appropriate format and at an appropriate pace depending on the user's emotional state, resulting in a more user-friendly experience.

[1403] An "instruction manual" is a document provided to explain how to use or operate a product or service.

[1404] A "document" is a collection of text or information written to convey a specific message.

[1405] "Analysis" is the process of examining data or information in detail to understand its structure and meaning.

[1406] A "generative AI model" is a model that uses artificial intelligence technology to generate new text or data.

[1407] A "summary" is a concise summary of the important parts of a document or data.

[1408] A "video format" is a data format that expresses movement by playing back a series of images or videos over time.

[1409] An "audio format" is a format in which sound or voice is represented as digital data.

[1410] "Cloud storage" is a remote data storage service that allows you to store and access data over the Internet.

[1411] An "emotion engine" is a technology / system that analyzes a user's emotional state in real time and utilizes the results.

[1412] A "user terminal" is hardware or a device that a user directly operates to input or receive information.

[1413] "Tone" refers to the tone, timbre, and atmosphere of a voice or text.

[1414] "Speed" refers to the speed at which information or content is provided to users.

[1415] The present invention is a system for analyzing an instruction manual, summarizing the contents thereof, and providing the summarization to a user. A specific embodiment of this system will be described below.

[1416] System Overview

[1417] Hardware Configuration

[1418] This system consists of three main components: user terminals, servers, and cloud storage. User terminals are devices such as PCs, smartphones, and tablets, while servers are computing devices for analysis and generation processing. Cloud storage is a remote data storage service that stores generated content and provides it to users.

[1419] Software Configuration

[1420] The system includes a PDF parsing library, a generative AI model (e.g., GPT-4), a text-to-speech engine (e.g., Google Text-to-Speech), video generation software, and an emotion engine.

[1421] Processing flow

[1422] User Operation

[1423] Users use a web browser or mobile application to upload instruction manual files (PDF or text files) to the system. Specifically, they access the system, click the "Upload" button, and select the instruction manual they want to upload from the file selection dialog.

[1424] Terminal operation

[1425] The device reads the instruction manual file selected by the user, converts it into binary data, and then sends the file data to the server as an HTTP POST request.

[1426] Server Processing

[1427] The server receives the file sent from the device and analyzes its contents. For PDF files, it extracts the text using a PDF analysis library, while for text files, it reads the text data directly. It then summarizes the analyzed text using a generative AI model. The generative AI model used (e.g., GPT-4) is given a prompt such as "Please briefly summarize the key points of this instruction manual" to generate an appropriate summary.

[1428] Content generation

[1429] The server inputs the summarized text into a text-to-speech engine to generate an audio file, and then uses video generation software to generate an animated video based on the summarized content, incorporating text slides and visuals and the generated audio file.

[1430] Emotion Engine Operation

[1431] The emotion engine recognizes the user's emotional state in real time. For example, if the user is using a webcam, it uses facial expression recognition technology to analyze their emotions and sends the results to the server. Based on the results, the server adjusts the content of the summary text and the tone and speed of the generated content.

[1432] Content provision

[1433] The server uploads the generated video and audio files to cloud storage and provides a download link or streaming URL to the user's device. The user uses this link to watch the video or audio. During viewing, the emotion engine monitors the user's reactions in real time and adjusts the content as needed.

[1434] Specific examples

[1435] For example, consider a user uploading a user manual for a home appliance (e.g., a smartphone). The user accesses a web application, clicks the "Upload" button, and selects the PDF file of the smartphone's user manual. The device sends this file to a server, which extracts text from the PDF and uses a generative AI model to generate a summary of key sections. The server then inputs the summarized text into a text-to-speech engine to generate an audio file. It then uses video generation software to create an animated video. During this process, an emotion engine recognizes the user's emotions and adjusts the summary content, tone, and speed of the video as needed. Finally, the server uploads the generated video and audio files to cloud storage and provides the user with a download link. The user clicks the link to watch the video and learn how to set up the smartphone and use its features.

[1436] Prompt Sentence Examples

[1437] "Please briefly summarize the important points of this instruction manual."

[1438] The above is a detailed description of the embodiment of the present invention. This system can convert complex instruction manuals into a user-friendly format and provide them adaptively using an emotion engine.

[1439] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1440] Step 1:

[1441] The user uploads an instruction manual to the system. The user accesses the system, clicks the "Upload" button using a web browser or mobile application, and selects the instruction manual file (PDF or text file) from the file selection dialog. The input is the instruction manual file, and the output is the path to the selected file.

[1442] Step 2:

[1443] The terminal reads the selected file and converts it to binary data. The terminal uses the file system API to read the specified file and converts it to binary data. It then sends this binary data to the server as an HTTP POST request. The input is the path of the file selected by the user, and the output is the binary data sent to the server.

[1444] Step 3:

[1445] The server receives the binary data sent from the terminal, saves the file, and analyzes the contents. The server first saves the file in a temporary directory. Next, if it is a PDF file, it uses a PDF parsing library to extract the text, or if it is a text file, it reads the text data directly. The input is binary data, and the output is the extracted plain text. Specific operations include calling the API of the PDF parsing library to extract the text from the file.

[1446] Step 4:

[1447] The server summarizes the extracted text using a generative AI model. The server preprocesses the extracted plain text to remove unnecessary white space and special characters. The generative AI model is then given a prompt sentence, "Please briefly summarize the important points of this instruction manual," to generate summarized text. The input is the parsed plain text, and the output is the summarized text.

[1448] Step 5:

[1449] The server converts the summarized text into audio format. The server inputs the summarized text into a text-to-speech engine (e.g., Google Text-to-Speech) to generate an audio file. The input is the summarized text, and the output is an audio file. Specifically, the Google Text-to-Speech API is used to convert text into audio.

[1450] Step 6:

[1451] The server generates a video that includes an audio file. The server uses video generation software to create an animated video by integrating text slides and visuals with the generated audio file. The input is the summarized text and audio file, and the output is the generated video file. Specifically, the video generation software runs a script to generate the video.

[1452] Step 7:

[1453] The emotion engine recognizes the user's emotional state in real time. If the user is using a webcam, the emotion engine uses facial expression recognition technology to analyze emotions and sends the results to the server. The input is the video feed from the webcam, and the output is the analyzed emotion data. Specifically, it runs a facial expression recognition algorithm to analyze the user's emotions.

[1454] Step 8:

[1455] The server adjusts the content based on the output of the emotion engine. The server uses the emotion data to adjust the content of the summary text and the tone and speed of the generated video and audio. The input is emotion data, and the output is the adjusted content. Specifically, the server changes the parameters of the generation algorithm in real time.

[1456] Step 9:

[1457] The server uploads the generated video and audio files to cloud storage. The server uploads the files using a cloud storage service and provides an access link to the user's device. The input is the generated content file, and the output is the upload result to cloud storage and the access link. Specifically, the server uses the cloud storage API to upload the files and generate a link.

[1458] Step 10:

[1459] The user views the content using the provided link. The user clicks on a download link or streaming URL to view the video or audio. The input is the access link, and the output is the content to be viewed. The specific operation is to play the content using a browser or media player.

[1460] The above is a detailed flow of the program processing for this system. The input and output at each step are clearly stated, and the type of data processing and calculation that is performed is specifically explained.

[1461] (Application example 2)

[1462] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1463] In recent years, many factories have introduced new robots, but complicated instruction manuals hinder workers' quick understanding, resulting in a decline in work efficiency. Furthermore, there is a lack of technology to adapt instructions according to the user's emotional state, which often leaves workers confused.

[1464] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing the text of the instruction manual, means for summarizing the analyzed text using a generative AI model, means for converting the summarized text into video and audio format, means for providing the converted content to a user terminal, means for recognizing the user's emotional state in real time, and means for adjusting the content of the summarized text and the tone and speed of the video and audio based on the user's emotional state. This makes it possible to improve work efficiency and reduce user confusion.

[1465] An "instruction manual" is a document that contains information on how to use, maintain, and handle a product.

[1466] "Analyzing" means breaking down given data and examining each element in detail.

[1467] A "generative AI model" is a model that uses artificial intelligence techniques to generate new data, such as summarizing or automatically generating text.

[1468] To "summarize" means to summarize a longer piece of text or information in a concise form.

[1469] "Video format" is a format that expresses information as moving images.

[1470] "Audio format" is a format in which information is conveyed auditorily.

[1471] A "user terminal" is a device such as a computer or smartphone that is directly operated by a user.

[1472] "Emotional state" refers to a user's current psychological state or mood.

[1473] "Real-time" is a term that refers to processing or response occurring immediately, as in actual time.

[1474] "Tone" is an expression that refers to the tone or color of a sound or voice.

[1475] "Speed" refers to the speed at which information is transmitted within a given period of time.

[1476] This system summarizes the complex content of instruction manuals, converts them into video and audio formats, and adaptively adjusts the content according to the user's emotions, allowing factory workers to easily understand the instruction manuals for newly installed robots and improving work efficiency.

[1477] System configuration

[1478] 1. User Operation

[1479] First, the user uploads the instruction manual (PDF or text file) using the smartphone app. Click the "Upload" button in the app and select the file to upload from the file selection dialog.

[1480] 2. Terminal operation

[1481] The uploaded file is read by the smartphone and converted into binary data, which is then sent to the server as an HTTP POST request.

[1482] 3. Server Processing

[1483] The server uses a PDF parsing library (e.g., fitz) to extract text from the PDF file, or reads it directly in the case of text files.

[1484] Then, a generative AI model (e.g., GPT-4) is used to extract key points from the plain text and generate a concise summary. A text-to-speech engine (e.g., gTTS) is then used to generate an audio file, which is then used to create an animated video using video generation software (e.g., moviepy).

[1485] 4. Emotion engine integration

[1486] The emotion engine recognizes the user's emotional state in real time through the user's camera feed. For example, if the user is confused, the server will adjust the summary, tone and speed of the video to provide a more detailed and slower explanation.

[1487] 5. Content Submissions and Feedback

[1488] The generated video and audio files are uploaded to cloud storage, and a download link or streaming URL is generated for the user. The user can use the link to watch the video or audio. The emotion engine monitors the user's reactions in real time while watching and adjusts the content as needed.

[1489] Specific examples

[1490] For example, a factory worker uploads a PDF instruction manual for a new robot to the app. The server extracts the text from the PDF and uses a generative AI model to summarize the key points of the manual. The summarized text is then fed into a text-to-speech engine to generate an audio file, which is then used by video generation software to create an animated video.

[1491] The emotion engine monitors the user's emotional state in real time and regenerates the summary content if they are confused, providing more detailed and slower explanations.

[1492] Prompt Sentence Examples

[1493] Summarize the following text:

[1494] [Text extracted from PDF]

[1495] It should be around 100-200 characters in length, focus on the key points, and be concise.

[1496] As described above, this system contributes to improving work efficiency by converting complex instruction manuals into an easily understandable format and providing adaptive explanations using an emotion engine.

[1497] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1498] Step 1:

[1499] The user uploads an instruction manual (PDF or text file) using a smartphone app. The input data is the instruction manual file on the user's device, and the output data is binary data sent to the server.

[1500] Step 2:

[1501] The terminal reads the uploaded file, converts it to binary data, and sends the converted binary data to the server as an HTTP POST request. The input is the file selected by the user, and the output is the binary data sent to the server.

[1502] Step 3:

[1503] The server parses the received binary data and extracts the text from the PDF file using a PDF parsing library (e.g., fitz) or reads the data directly from the text file. The input is the binary data sent to the server, and the output is the extracted text data.

[1504] Step 4:

[1505] The server generates a summary from the plain text using a generative AI model (e.g., GPT-4). In this process, the input is the extracted text data, and the output is the generated summary text. Specifically, a prompt sentence is input to the generative AI model, and the model performs a process of summarizing the key points.

[1506] Step 5:

[1507] The server inputs the generated summary text into a text-to-speech engine (e.g., gTTS) to generate an audio file. The input is the summarized text data, and the output is an audio file.

[1508] Step 6:

[1509] The server creates an animated video using video generation software (e.g., moviepy) based on the generated audio file. The input is the audio file and summary text, and the output is the generated animated video file.

[1510] Step 7:

[1511] The emotion engine recognizes the user's emotional state in real time through the user's camera feed. The input is data from the user's camera feed, and the output is data on the user's emotional state. Based on the emotional state, it adjusts the tone and speed of the video and audio.

[1512] Step 8:

[1513] The server realigns the summary text and the generated video and audio files according to the output data of the emotion engine. The input is the emotion engine data, and the output is the realigned content.

[1514] Step 9:

[1515] Finally, the generated video and audio files are uploaded to cloud storage, and a download link or streaming URL is provided to the user's device. The input is the generated video and audio files, and the output is uploading them to cloud storage and providing a link to the user.

[1516] A concrete example would be a factory floor worker uploading a new robot instruction manual to an app, watching abridged video and audio, and receiving detailed, slow-fire explanations when the emotion engine detects confusion.

[1517] Example prompt sentence:

[1518] Summarize the following text:

[1519] [Text extracted from PDF]

[1520] It should be around 100-200 characters in length, focus on the key points, and be concise.

[1521] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1522] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1523] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1524] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1525] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1526] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1527] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1528] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1529] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1530] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1531] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1532] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1533] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1534] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1535] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1536] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1537] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1538] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1539] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1540] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1541] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1542] The following is further disclosed regarding the above embodiment.

[1543] (Claim 1)

[1544] means for analyzing the text of the instruction manual;

[1545] A means for summarizing the analyzed text using a generative AI model; and

[1546] means for converting the summarized text into video and audio formats;

[1547] means for providing the converted content to a user terminal;

[1548] A system including:

[1549] (Claim 2)

[1550] 10. The system of claim 1, further comprising means for removing redundant information from the analyzed text.

[1551] (Claim 3)

[1552] 10. The system of claim 1, further comprising means for converting the summarized text into an audio file using a text-to-speech engine.

[1553] "Example 1"

[1554] (Claim 1)

[1555] a means for analyzing the instruction manual data;

[1556] A means for summarizing the analyzed data using a generative AI model; and

[1557] means for converting the summarized data into audio and video formats;

[1558] means for providing the converted content to a user's terminal;

[1559] A system including:

[1560] (Claim 2)

[1561] 10. The system of claim 1, further comprising means for filtering out redundant information from the analyzed data.

[1562] (Claim 3)

[1563] 10. The system of claim 1, further comprising means for converting the summarized data into an audio file using a text-to-speech engine.

[1564] "Application Example 1"

[1565] (Claim 1)

[1566] means for analyzing the text of the instruction manual;

[1567] A means for summarizing the analyzed text using a generative AI model; and

[1568] means for converting the summarized text into video and audio formats;

[1569] means for providing visual and audio summaries to a user via smart glasses;

[1570] means for providing the converted content to a user terminal;

[1571] A system including:

[1572] (Claim 2)

[1573] 10. The system of claim 1, further comprising means for removing redundant information from the analyzed text.

[1574] (Claim 3)

[1575] 10. The system of claim 1, further comprising means for converting the summarized text into an audio file using a text-to-speech engine.

[1576] (Claim 4)

[1577] 10. The system of claim 1, further comprising means for converting the summarized text into an animated video using video generation software.

[1578] (Claim 5)

[1579] 10. The system of claim 1, further comprising means for providing the contents of an instruction manual for the factory robot in real time in cooperation with the smart glasses.

[1580] "Example 2: Combining Emotion Engines"

[1581] (Claim 1)

[1582] means for analyzing the instruction manual document;

[1583] A means for summarizing the analyzed documents using a generative AI model; and

[1584] means for converting the summarized documents into video and audio formats;

[1585] means for uploading the converted content to a cloud storage for provision to a user terminal;

[1586] a means for recognizing a user's emotional state using an emotion engine to adjust the summary, tone, and speed of the content;

[1587] A system including:

[1588] (Claim 2)

[1589] 10. The system of claim 1, further comprising means for removing unnecessary data from the analyzed document.

[1590] (Claim 3)

[1591] 10. The system of claim 1, further comprising means for using a text-to-speech engine to convert the summarized document into an audio file.

[1592] "Application example 2 when combining emotion engines"

[1593] (Claim 1)

[1594] means for analyzing the text of the instruction manual;

[1595] A means for summarizing the analyzed text using a generative AI model; and

[1596] means for converting the summarized text into video and audio formats;

[1597] means for providing the converted content to a user terminal;

[1598] means for recognizing a user's emotional state in real time;

[1599] means for adjusting the content of the summary text and the tone and speed of the video and audio based on the emotional state of the user;

[1600] A system including:

[1601] (Claim 2)

[1602] 10. The system of claim 1, further comprising means for removing redundant information from the analyzed text.

[1603] (Claim 3)

[1604] 10. The system of claim 1, further comprising means for converting the summarized text into an audio file using a text-to-speech engine. [Explanation of symbols]

[1605] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for analyzing the text of the instruction manual; A means for summarizing the analyzed text using a generative AI model; and means for converting the summarized text into video and audio formats; means for providing the converted content to a user terminal; A system including:

2. The system of claim 1 further comprising means for removing redundant information from the analyzed text.

3. The system of claim 1 further comprising means for converting the summarized text into an audio file using a text-to-speech engine.

Images