System

Abstract

To provide a system.SOLUTION: The system includes: means for recognizing the operational procedure of a computer; means for automatically generating a screen shot and a text on the basis of the operational procedure; means for preparing a manual by using the generated screen shot and the text; means for recognizing the emotion of the user; means for adjusting the method for distributing the generated manual on the basis of the recognized emotion; and providing the generated manual in plural output formats.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Creating manuals takes time and there is room for improvement in efficiency. In particular, recognizing operation procedures, generating screenshots and text, and creating tables of contents and indexes are often done manually, so the goal is to improve the efficiency of manual creation by automating these tasks. [Means for solving the problem]

[0005] The present invention provides a means for recognizing computer operating procedures and automatically generating screenshots and text based on the operating procedures. It also provides a means for creating manuals using the generated screenshots and text, and has a function for automatically generating a table of contents and index for the manual. It also has a function for providing the generated manual in various output formats, facilitating the distribution and sharing of the manual. This realizes efficient manual creation. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] The present invention is a manual creation assistant system. This system recognizes the operations a user performs on a computer and automatically generates screenshots and text corresponding to those operations. Specifically, when a user operates a specific piece of software, the system recognizes the operating procedure and generates corresponding screenshots and explanatory text. This information forms the basis for creating a manual.

[0029] "Example 2"

[0030] Furthermore, this embodiment of the present invention has a function to automatically create a manual using the generated screenshots and text. Specifically, the generated screenshots and text are combined, organized, and compiled into a manual format. It also has a function to automatically generate a table of contents and index for the manual. This makes it possible to significantly reduce the effort required to create a manual.

[0031] "Example 3"

[0032] Furthermore, the present invention has the function of providing the generated manual in various output formats. Specifically, the manual can be output in a format required by the user, such as PDF, HTML, or Word document. This makes it easy to distribute and share the manual.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: The user operates a specific piece of software.

[0036] Step 2: The system recognizes the user's operating procedure.

[0037] Step 3: The system will generate screenshots and explanatory text corresponding to the operating steps.

[0038] "Example 2"

[0039] Step 1: Get the screenshots and text generated by the system.

[0040] Step 2: The system combines this information and compiles it into a manual format.

[0041] Step 3: The system automatically generates a table of contents and index for the manual.

[0042] "Example 3"

[0043] Step 1: Obtain the system-generated manual.

[0044] Step 2: The system outputs the manual in the format required by the user, such as PDF, HTML, Word document, etc.

[0045] Example 1

[0046] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0047] Traditional manual creation required manually recording operation procedures, taking screenshots, and creating explanatory text, which was extremely time-consuming and labor-intensive. Furthermore, there was a high possibility of human error when recording operation procedures and creating explanatory text. Furthermore, creating a table of contents and index for the generated manual, providing it in various output formats, and distributing and sharing it were also time-consuming and labor-intensive.

[0048] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0049] In this invention, the server includes a means for monitoring user operations, a means for recording the operation details, a means for taking screenshots in response to the operations, a means for generating explanatory text using a generative AI model, and a means for creating a manual using the generated screenshots and explanatory text. This allows for real-time monitoring of user operations, automatic generation of screenshots and explanatory text based on the operation procedures, and efficient and accurate creation of manuals. It also enables automatic generation of a table of contents and index for the manual, provision in various output formats, and easy distribution and sharing.

[0050] A "means for monitoring user operations" is a device or software that has the function of monitoring the operations performed by a user on a computer in real time and capturing operation events.

[0051] "Means for recording operation details" refers to a device or software that has the function of recording the operation details of the monitored user in a log file or database and saving them in a format that can be referenced later.

[0052] A "means for taking a screenshot" is a device or software that has the function of capturing an image of the screen at an appropriate time in response to a user's operation and saving it as an image file.

[0053] A "means for generating explanatory text using a generative AI model" is a device or software that has the function of automatically generating explanatory text in natural language using a generative AI model based on recorded operation content.

[0054] A "means for creating a manual" is a device or software that has the function of combining the generated screenshots with explanatory text to form each page of the manual and to create a complete manual including a table of contents and index.

[0055] The "function to automatically generate a table of contents and index" is a function that automatically generates a table of contents and index based on the contents of the manual, allowing users to easily search for the information they need.

[0056] "Ability to provide in various output formats" refers to the ability to output the generated manual in different file formats such as PDF, HTML, DOCX, etc., and provide it flexibly according to the user's needs.

[0057] "Functions that make distribution and sharing easy" refers to functions that allow the generated manuals to be easily distributed and shared via email, cloud storage, an internal network, etc.

[0058] MODE FOR CARRYING OUT THE INVENTION

[0059] This invention is a manual creation assistant system that recognizes user operations on a computer and automatically generates screenshots and explanatory text corresponding to those operations. This system monitors user operations in real time and automatically generates screenshots and explanatory text based on the operating procedures, enabling efficient and accurate manual creation.

[0060] Hardware and software used

[0061] 1. Terminal

[0062] A computer device that monitors and records user operations.

[0063] Specifically, it includes software that captures events such as user mouse clicks, keyboard input, and window switching.

[0064] 2. Server

[0065] A computer device that generates explanatory text based on recorded operations and screenshots to create manuals.

[0066] A generative AI model (e.g., OpenAI's GPT-4®) is used to create text that explains the operating procedures in natural language.

[0067] Data processing and calculation

[0068] 1. Monitor user actions

[0069] The terminal monitors the user's operations in real time when using specific software.

[0070] Specifically, the terminal captures events such as user mouse clicks, keyboard input, and window switches.

[0071] 2. Record your actions

[0072] The terminal records the monitored operation details in a log file.

[0073] Specifically, it includes the type of operation, a timestamp, and information about the window or application that is the target of the operation.

[0074] 3. Take a screenshot

[0075] The device will automatically take screenshots at appropriate times depending on the user's actions.

[0076] Specifically, a screenshot is captured immediately after the operation is completed and saved as an image file.

[0077] 4. Generate explanatory text

[0078] The server generates explanatory text based on the recorded operations and screenshots.

[0079] Use a generative AI model (e.g., OpenAI's GPT-4) to create text that explains operating procedures in natural language.

[0080] 5. Configuring the manual

[0081] The server combines the acquired screenshots with the generated explanatory text to create each page of the manual.

[0082] Specifically, screenshots and corresponding explanatory text are laid out in HTML or PDF format and saved as a manual.

[0083] Specific examples

[0084] The user performs image editing operations using image processing software.

[0085] The device monitors and records when a user opens the "Filter" menu and selects the "Blur" option.

[0086] Your device will take a screenshot of the moment you select the "Blur" option.

[0087] Based on the recorded actions and screenshots, the server generates the following explanatory text: "Open the 'Filter' menu and select the 'Blur' option."

[0088] The server combines screenshots and explanatory text to form one page of the manual.

[0089] Prompt Sentence Examples

[0090] "Please explain the steps a user would take to edit an image in your image processing software, including applying a filter."

[0091] In this way, manuals can be created efficiently by monitoring user operations and automatically generating screenshots and explanatory text.

[0092] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0093] Step 1:

[0094] Monitor user actions

[0095] The terminal monitors the user's operations in real time when using specific software.

[0096] Input: User interaction events such as mouse clicks, keyboard input, and window switching.

[0097] Data processing: Capture operation events and extract information about the operation type, timestamp, and the window or application being operated on.

[0098] Output: The captured operation event data.

[0099] Specific behavior: Monitors when a user clicks the "New" button in Microsoft (registered trademark) Word.

[0100] Step 2:

[0101] Record your operations

[0102] The terminal records the monitored operation details in a log file.

[0103] Input: The captured operation event data.

[0104] Data processing: Writes operation event data to a log file.

[0105] Output: A log file containing the operations.

[0106] Specific operation: The user clicking the "New" button is recorded as "2023-10-01 10:00:00 - Microsoft Word - Clicked the New button."

[0107] Step 3:

[0108] Take a screenshot

[0109] The device will automatically take screenshots at appropriate times depending on the user's actions.

[0110] Input: The captured operation event data.

[0111] Data Processing: Capture screenshots immediately after the operation is completed and save them as image files.

[0112] Output: Image files of the captured screenshots.

[0113] Specific behavior: Capture the screen immediately after the user clicks the "Create New" button and save it as "screenshot_20231001_100000.png".

[0114] Step 4:

[0115] Generate descriptive text

[0116] The server generates explanatory text based on the recorded operations and screenshots.

[0117] Input: Log file recording operation details, image file of captured screenshots.

[0118] Data processing: Generative AI models (e.g., OpenAI's GPT-4) are used to create text that explains operating procedures in natural language.

[0119] Output: The generated description text.

[0120] Specific operation: In response to the action of clicking the "Create New" button, the server generates the text "Click the New button in Microsoft Word to create a new document."

[0121] Step 5:

[0122] Configuring the manual

[0123] The server combines the acquired screenshots with the generated explanatory text to create each page of the manual.

[0124] Input: Image files of captured screenshots, generated description text.

[0125] Data processing: Screenshots and corresponding explanatory text are laid out in HTML or PDF format and saved as a manual.

[0126] Output: A file with the configured manual.

[0127] Specific operation: The server combines "screenshot_20231001_100000.png" and the text "Click the New button in Microsoft Word to create a new document." into one page.

[0128] (Application example 1)

[0129] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0130] Conventional manual creation methods require manual recording of operating procedures, taking screenshots, and creating explanatory text, which is time-consuming and labor-intensive. Furthermore, recording the operating procedures of factory robots and generating detailed explanations is even more complex and requires specialized knowledge, making efficient manual creation difficult. This makes it difficult to accurately record and share operating procedures, resulting in reduced work efficiency and an increased risk of operational errors.

[0131] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0132] In this invention, the server includes means for recognizing computer operation procedures, means for automatically generating screenshots and text based on the operation procedures, means for creating a manual using the generated screenshots and text, means for recording robot operation procedures and generating screenshots and explanatory text, and means for generating detailed explanations of the operation procedures using a generative AI model, thereby enabling automatic recording of operation procedures and generation of detailed explanations.

[0133] "Means for recognizing computer operation procedures" is a function for detecting the operations performed by a user on a computer and recording those procedures.

[0134] "Means for automatically generating screenshots and text" is a function that automatically creates screen images and their descriptions based on recognized operating procedures.

[0135] The "means for creating a manual" is a function for creating a document explaining an operating procedure using the generated screenshots and text.

[0136] The "means for recording the robot's operation procedures" is a function for detecting a series of operations performed by a factory robot and recording those procedures.

[0137] The "means for generating explanatory text" is a function that automatically creates text that explains a procedure based on a recorded operating procedure.

[0138] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate text or images.

[0139] The "means for generating detailed explanations of operating procedures" is a function that uses a generative AI model to automatically create detailed explanations for recorded operating procedures.

[0140] The system for implementing this invention has the function of recognizing computer operating procedures, automatically generating screenshots and text, and creating manuals. It also includes the function of recording operating procedures of factory robots and generating detailed instructions using a generative AI model.

[0141] The server uses software to detect user actions on the computer. This software monitors user actions in real time and records the user actions. Based on the recorded actions, it captures screenshots and generates corresponding explanatory text.

[0142] The server then automatically creates a manual that explains the operating procedures using the generated screenshots and text. This manual is structured to be easily understandable by the user, and a table of contents and index are also automatically generated.

[0143] Furthermore, to record the operating procedures of the factory robot, the server monitors the robot's movements and records the procedures. Based on the recorded procedures, screenshots and explanatory text are generated. A generative AI model is used to automatically create detailed descriptions of the operating procedures. This generative AI model is implemented using OpenAI's API.

[0144] As a concrete example, consider the case of recording the operation procedure for a robot to pick up part A and attach it to part B. The server monitors the robot's operation and captures a screenshot of the moment when part A is picked up. Next, using the generative AI model, it inputs the prompt sentence "Please generate a detailed explanation of the procedure for picking up part A and attaching it to part B," and generates a detailed explanatory text.

[0145] Example prompt sentence:

[0146] Robot operation: Pick up part A and attach it to part B. Generate a detailed description of this operation.

[0147] In this way, it becomes possible to automatically record operating procedures and generate detailed explanations, thereby realizing efficient manual creation.

[0148] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0149] Step 1:

[0150] The server recognizes the actions that users take on their computers.

[0151] Input: User actions

[0152] Data processing: detecting and recording operating procedures

[0153] Output: Recorded operation steps

[0154] Specific operation: The server uses software that monitors the operation procedure in real time and records the operations performed by the user.

[0155] Step 2:

[0156] The server captures screenshots based on the recorded operating procedures.

[0157] Input: Recorded operating procedures

[0158] Data processing: Screenshot capture

[0159] Output: Captured screenshot

[0160] Specific operation: The server captures images of the screen corresponding to each step of the operating procedure and saves them as screenshots.

[0161] Step 3:

[0162] The server generates explanatory text based on the recorded operating procedures.

[0163] Input: Recorded operating procedures

[0164] Data processing: generating explanatory text

[0165] Output: Generated description text

[0166] Specific operation: The server automatically generates explanatory text corresponding to each step of the operating procedure and saves it as text.

[0167] Step 4:

[0168] The server creates a manual using the generated screenshots and text.

[0169] Input: Screenshot, Description text

[0170] Data processing: Manual creation

[0171] Output: Created manual

[0172] Specific operation: The server automatically creates a document that explains the operation procedure by combining screenshots and explanatory text.

[0173] Step 5:

[0174] The server records the operating procedures of the factory robot.

[0175] Input: Robot control

[0176] Data processing: detecting and recording operating procedures

[0177] Output: Recorded operation steps

[0178] Specific operation: The server monitors the robot's operation and records the operation procedures in real time.

[0179] Step 6:

[0180] The server generates screenshots and explanatory text based on the recorded robot operation procedures.

[0181] Input: Recorded operating procedures

[0182] Data processing: Capture screenshots and generate explanatory text

[0183] Output: Captured screenshots, generated explanatory text

[0184] Specific operation: The server captures screen images corresponding to each step of the robot's operating procedure and generates detailed explanatory text using a generative AI model.

[0185] Step 7:

[0186] The server uses a generative AI model to generate detailed instructions for the operation procedure.

[0187] Input: Prompt for operation procedure

[0188] Data processing: Text generation using generative AI models

[0189] Output: Detailed description text

[0190] Specific operation: The server uses a generative AI model (e.g., OpenAI API) to generate detailed instructions for the operation procedure. Example prompt: "Robot operation procedure: Pick up part A and attach it to part B. Please generate a detailed explanation for this operation."

[0191] Step 8:

[0192] The server uses the generated screenshots and detailed explanatory text to create a manual that explains how to operate the robot.

[0193] Input: screenshots, detailed description text

[0194] Data processing: Manual creation

[0195] Output: Created manual

[0196] Specific operation: The server automatically creates a document that explains the robot's operating procedures by combining screenshots and detailed explanatory text.

[0197] Example 2

[0198] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0199] The traditional manual creation process requires a lot of time and effort, especially when it comes to organizing screenshots and text and creating tables of contents and indexes. It's also difficult to maintain the quality and consistency of the generated manuals. This creates a demand for efficient, high-quality manual creation.

[0200] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for a user to input data, a means for generating screenshots and text using a generative AI model, a means for creating a manual by combining the generated screenshots and text, a means for automatically generating a table of contents and index for the manual, and a means for outputting the completed manual in PDF format and providing it to the user. This enables efficient and high-quality manual creation.

[0201] The "means for the user to input data" is an interface that allows the user to provide the system with the information required to create the manual.

[0202] "Means for generating screenshots and text using a generative AI model" refers to a function that uses a generative AI model to automatically generate screenshots and text from data provided by a user.

[0203] The "means for creating a manual by combining the generated screenshots and text" is a function for organizing the generated screenshots and text and putting them together in the format of a manual.

[0204] "Means for automatically generating a table of contents and index for a manual" is a function that extracts the titles of each chapter and section in a manual, creates a table of contents along with page numbers, and extracts important keywords to create an index.

[0205] "Means for outputting the completed manual in PDF format and providing it to the user" refers to a function for saving the final created manual in PDF format and providing it to the user so that it can be downloaded.

[0206] "A function to adjust screenshots to an appropriate size using image processing software" refers to a function that uses software to process screenshots to an appropriate size and resolution.

[0207] "The ability to format text using text editing software" is the ability to use software to format the generated text into an appropriate format.

[0208] "A function that uses natural language processing technology to extract important keywords, list the pages on which those keywords appear, and create an index" is a function that uses natural language processing technology to extract important keywords from text, list the pages on which those keywords appear, and create an index.

[0209] MODE FOR CARRYING OUT THE INVENTION

[0210] This invention is a system that allows users to input data, generates screenshots and text using a generative AI model, and combines them to create a manual. It also has the ability to automatically generate a table of contents and index for the manual and output the completed manual in PDF format.

[0211] 1. Program Generation

[0212] A user accesses the system's interface using a web browser. On the interface, the user enters a prompt and uploads the necessary screenshots. For example, the user can enter the prompt "Please create a manual that explains the installation procedure for new software. Please use the screenshots and text below." and upload screenshots of the installation procedure.

[0213] 2. Program Processing

[0214] The server receives the prompt text entered by the user and invokes a generative AI model (e.g., OpenAI's GPT-4). The generative AI model generates appropriate text based on the prompt text and processes images based on screenshots uploaded by the user. For example, it generates instructions for each step of the installation procedure.

[0215] The server then combines the generated screenshots with the text, using image processing software to scale the screenshots to the appropriate size and text editing software to format the text, placing the screenshot on each page with the corresponding description below it.

[0216] The server also automatically generates a table of contents and index for the manual. Using natural language processing (NLP) technology, it extracts the titles of each chapter and section and creates a table of contents along with page numbers. It also extracts important keywords and creates an index by listing the pages where those keywords appear. For example, it extracts keywords such as "install," "configure," and "complete," and organizes the pages where those keywords appear into an index.

[0217] Finally, the server outputs the completed manual in PDF format. The PDF file is provided as a downloadable link to the user. The user clicks this link to download the manual, which can then be printed or shared as needed. For example, a user clicks the link to download the PDF file and share it with a colleague.

[0218] 3. Specific Examples

[0219] As a concrete example, consider a case where a user wants to create a manual for the installation procedure of new software. The user inputs screenshots and descriptions of the installation procedure into a generative AI model.

[0220] Example prompt sentence:

[0221] "Please create a manual that explains the installation process for new software. Use the following screenshots and text."

[0222] By inputting this prompt into a generative AI model, the model can automatically generate a detailed installation manual based on screenshots and text, including screenshots and corresponding instructions for each step, as well as an automatically generated table of contents and index.

[0223] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0224] Step 1:

[0225] The user enters data. Specifically, the user accesses the system interface using a web browser, enters a prompt, and uploads the necessary screenshots. The entered prompt and screenshots are sent to the server. For example, the user might enter a prompt such as "Please create a manual explaining the installation procedure for new software. Please use the screenshots and text below," and upload screenshots of the installation procedure.

[0226] Step 2:

[0227] The server generates screenshots and text using a generative AI model. The server receives the prompt text entered by the user and calls a generative AI model (e.g., GPT-4). The generative AI model generates appropriate text based on the prompt text and processes the image based on the screenshot uploaded by the user. The output is the generated text and the processed screenshot. For example, it generates instructions corresponding to each step of the installation procedure.

[0228] Step 3:

[0229] The server combines the generated screenshots with the text to create the manual. It uses image processing software to scale the screenshots to the appropriate size and text editing software to format the text. Specifically, it places a screenshot on each page and places corresponding explanatory text below it. The output is the initial version of the manual.

[0230] Step 4:

[0231] The server automatically generates a table of contents and index for the manual. Using natural language processing (NLP) technology, the server extracts the titles of each chapter and section and creates a table of contents along with page numbers. It also extracts important keywords and creates an index by listing the pages where these keywords appear. The output is a manual with an added table of contents and index. For example, keywords such as "installation," "configuration," and "completion" are extracted, and the pages where these keywords appear are organized into an index.

[0232] Step 5:

[0233] The server outputs the completed manual in PDF format and provides it to the user. The server saves the final manual in PDF format and provides it as a download link for the user. The user clicks this link to download the manual and can print or share it as needed. As output, a download link is provided to the user. For example, the user clicks the link to download the PDF file and share it with a colleague.

[0234] (Application example 2)

[0235] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0236] Conventionally, recording and creating manuals for operating and maintenance procedures for factory robots has relied on manual work, which has been time-consuming and labor-intensive. Creating the manual's table of contents and indexes has also been done manually, which has been inefficient and prone to errors. Furthermore, there has been a lack of a way to provide the generated manuals in various output formats for easy distribution and sharing.

[0237] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0238] In this invention, the server includes means for recognizing computer operating procedures, means for automatically generating screenshots and text based on the operating procedures, means for creating manuals using the generated screenshots and text, means for providing the generated manuals in various output formats to facilitate distribution and sharing of the manuals, and means for automatically recording factory robot operating procedures and maintenance procedures. This automates the recording of factory robot operating procedures and maintenance procedures and manual creation, significantly reducing time and labor. Furthermore, automatic generation of tables of contents and indexes enables efficient manual creation with fewer errors. Furthermore, providing the generated manuals in various output formats facilitates distribution and sharing, enabling information to be transmitted quickly and accurately.

[0239] "Means for recognizing computer operating procedures" refers to a device or software that monitors computer operations and automatically identifies those operating procedures.

[0240] A "screenshot" is a technique for capturing an image of what is displayed on a computer screen.

[0241] "Means for automatically generating text" refers to a device or software for automatically generating explanatory text or instruction text based on operating procedures.

[0242] The "means for creating a manual" refers to a device or software for combining the generated screenshots with text to create a document that summarizes operating procedures and explanations.

[0243] "Various output formats" refers to ways to store and present information in different file formats, such as PDF, HTML, DOCX, etc.

[0244] The "means for facilitating distribution and sharing of manuals" refers to a device or software for easily distributing and sharing the generated manuals via email, cloud storage, etc.

[0245] A "factory robot" is a mechanical device used to automate work in a factory.

[0246] "Means for automatically recording operating procedures and maintenance procedures" refers to a device or software that automatically records the operation and maintenance processes of factory robots and makes them available for later reference.

[0247] The "function to automatically generate a table of contents and index" is a function that automatically extracts each item and keyword in a manual and creates a table of contents and index.

[0248] A system for implementing this invention automatically records the operating procedures and maintenance procedures of factory robots and automatically generates manuals using screenshots and text. A specific embodiment of this system will be described below.

[0249] Hardware and software used

[0250] Hardware: Factory robots, cameras, computers

[0251] Software: Python, PIL (Python Imaging Library), FPDF, pytesseract

[0252] Data processing and calculation

[0253] A means of recognizing computer operating procedures

[0254] To recognize the operating procedures of the factory robot, the server uses a camera to capture images of the robot's operating screen, which are then sent to a computer where the operating procedures are automatically identified.

[0255] A way to automatically generate screenshots and text

[0256] The server saves the captured image as a screenshot and automatically generates instructions and instructions based on the operation procedure by extracting text from the image using pytesseract and generating appropriate instructions using a generative AI model.

[0257] How to create a manual

[0258] The server combines the generated screenshots and text to create a manual that summarizes operation procedures and explanations. To do this, it uses FPDF to generate the manual in PDF format.

[0259] A means to provide a variety of output formats for easy distribution and sharing

[0260] The server saves the generated manuals in various output formats such as PDF, HTML, and DOCX for distribution and sharing via email or cloud storage.

[0261] A means of automatically recording operating and maintenance procedures for factory robots

[0262] The server automatically records the operation and maintenance processes of factory robots and makes them available for future reference, ensuring that operation and maintenance procedures are accurately recorded and reflected in manuals.

[0263] Specific examples

[0264] For example, when initializing a factory robot, the server uses a camera to capture a screenshot of the operation screen. Next, it uses pytesseract to extract text from the image and uses a generative AI model to automatically generate an explanatory text such as, "In this procedure, you will perform the initial setup of the robot." This information is then combined to generate a PDF manual using FPDF, which is then saved in cloud storage.

[0265] Prompt Sentence Examples

[0266] Use the "Robot Operation Manual Auto-Generation App" to add the following steps to the manual.

[0267] 1. Initial setup of the robot

[0268] 2. Calibrating the robot

[0269] In this way, factory robot operation and maintenance procedures can be efficiently recorded and manuals can be automatically generated, significantly reducing time and labor and enabling accurate and efficient manual creation.

[0270] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0271] Step 1:

[0272] The server takes a picture of the factory robot's operation screen with a camera and captures a screenshot. The input is the image taken by the camera, and the output is a screenshot image file. This image file is used in subsequent processing.

[0273] Step 2:

[0274] The server analyzes the captured screenshot image using pytesseract and extracts the text from the image. The input is the screenshot image file, and the output is the extracted text data. This text data is used to generate operating procedures and explanations.

[0275] Step 3:

[0276] The server uses a generative AI model to automatically generate appropriate explanations and instructions based on the extracted text data. The input is the extracted text data, and the output is the generated explanations and instructions. These explanations and instructions are used as the content of the manual.

[0277] Step 4:

[0278] The server combines the generated screenshots and explanatory text to create a PDF manual using FPDF. The input is the screenshot image file and the generated explanatory text, and the output is a PDF manual file. This manual file is used for subsequent distribution and sharing.

[0279] Step 5:

[0280] The server saves the generated PDF manuals in cloud storage and distributes them by email as needed. The input is the PDF manual file, and the output is saving to cloud storage and distributing by email. This makes it easy to distribute and share the manuals.

[0281] Step 6:

[0282] The server automatically records the operating procedures and maintenance procedures of the factory robots and makes them available for future reference. The input is the operation and maintenance process of the factory robots, and the output is the recorded operating and maintenance procedure data. This data is used as the contents of the manuals.

[0283] Step 7:

[0284] The server automatically generates a table of contents and index for the generated manual. The input is a PDF manual file, and the output is the final manual file with the table of contents and index added. This makes it easy to refer to the manual.

[0285] Example 3

[0286] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0287] Conventional manual creation systems require users to manually input content and format it, which takes time and effort. Furthermore, it is difficult to distribute or share the generated manuals, making efficient information transmission difficult. Furthermore, it is difficult to flexibly generate manuals that meet user requirements.

[0288] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0289] In this invention, the server includes a means for receiving a user's request, a means for inputting a prompt sentence into a generative AI model, a means for converting the generated manual into a specified format, and a means for providing the manual to the user, thereby enabling automatic generation of a manual according to a user's request and output in various formats.

[0290] "Means for receiving user requests" refers to an interface for sending prompts and instructions entered by the user to the system.

[0291] "Means for inputting prompt sentences into the generative AI model" refers to the function of inputting prompt sentences received from the user into the generative AI model in an appropriate format and performing processing to generate the contents of the manual.

[0292] "Means for converting the generated manual into a specified format" refers to a function for converting the contents of the manual output from the generative AI model into a format specified by the user (PDF, HTML, Word document, etc.).

[0293] The "means for providing the manual to the user" refers to a method for providing the generated manual to the user, and includes, for example, generating a download link or sending an email.

[0294] A "generative AI model" refers to an artificial intelligence model that generates text based on a user's prompt, such as a model that uses natural language processing technology.

[0295] A "prompt" is a piece of text containing instructions or requests that a user inputs to a generative AI model.

[0296] The present invention relates to a system for automatically generating a manual based on a user's request and providing it in a specified format. A specific embodiment of this system will be described below.

[0297] System configuration

[0298] This system mainly consists of three elements: a server, a terminal, and a user.

[0299] 1. Server:

[0300] The server receives requests from users, generates manuals using generative AI models, converts them into the specified format, and provides them. The server includes the following software and hardware:

[0301] Generative AI models: For example, models using natural language processing techniques (e.g., GPT-4) are used.

[0302] PDF generation libraries: For example, generate manuals in PDF format using LaTeX or ReportLab.

[0303] Web server: A web server (e.g., Apache (registered trademark), Nginx) that receives requests from users and returns processing results.

[0304] 2. Terminal:

[0305] A terminal is a device through which a user accesses the system and enters prompts. A terminal includes the following software and hardware:

[0306] Web Browser: A browser that allows users to access the system's web interface.

[0307] 3. User:

[0308] The user is responsible for inputting instructions to generate a manual using the system. The user accesses the system using a terminal and inputs a prompt.

[0309] System Operation

[0310] 1. Receive the user request:

[0311] The user accesses the system's web interface using a terminal and inputs a prompt statement, such as "I would like you to create a basic work procedure manual for new employees. The contents should include attendance procedures, basic work flows, and company rules. The output format should be PDF."

[0312] 2. Input the prompt sentence into the generative AI model:

[0313] The server analyzes the prompt received from the user and inputs it into a generative AI model (e.g., GPT-4), which then generates the content of the manual based on the prompt.

[0314] 3. Convert the generated manual to the specified format:

[0315] The server receives the manual content output from the generative AI model and converts it into the format specified by the user (PDF, HTML, Word document, etc.) For example, if the PDF format is specified, the server generates a PDF file using LaTeX or ReportLab.

[0316] 4. Provide the manual to users:

[0317] The server provides the generated manual to the user. Specifically, it generates a download link for the generated PDF file and notifies the user. The user then accesses the download link using their terminal and downloads the generated manual.

[0318] Specific examples

[0319] For example, if a user enters a prompt such as "I would like you to create a basic work procedure manual for new employees. The contents should include attendance procedures, basic work flow, and company rules. The output format should be PDF," the server will process as follows:

[0320] 1. Receiving the prompt:

[0321] User-entered prompt: "Please create a basic work procedure manual for new employees. The contents should include attendance procedures, basic work flow, and company rules. The output format should be PDF."

[0322] 2. Generate manuals using generative AI models:

[0323] The server inputs the prompt sentence into GPT-4 and generates the contents of the business procedure manual.

[0324] 3. Output the manual in the specified format:

[0325] The server embeds the generated text into a LaTeX template and generates a PDF file.

[0326] 4. Provide the manual to users:

[0327] The server sends the user a download link for the generated PDF file, which the user clicks to save in their local storage.

[0328] In this way, the system generates a manual in response to a user's request and provides it in the specified format. This makes it easy to distribute and share the manual. The flow of the specification process in the third embodiment will be described with reference to FIG. 15.

[0329] Step 1:

[0330] Receives user requests

[0331] The user uses a terminal to access the system's web interface and input a prompt. For example, the input prompt might be, "Please create a basic work procedure manual for new employees. Please include attendance procedures, basic work flow, and company rules. Please output in PDF format." The terminal then sends this prompt to the server.

[0332] Input: The prompt text entered by the user

[0333] Output: The prompt sent to the server

[0334] Step 2:

[0335] Input the prompt sentence into the generative AI model

[0336] The server analyzes the received prompt sentence and prepares it for input to the generative AI model. Specifically, the prompt sentence is formatted appropriately and input to the generative AI model (e.g., GPT-4). The generative AI model generates the contents of the manual based on the prompt sentence.

[0337] Input: The prompt sent to the server

[0338] Output: The prompts input to the generative AI model and the contents of the generated manual

[0339] Step 3:

[0340] Convert the generated manual to the specified format

[0341] The server receives the manual content output from the generative AI model and converts it into the format specified by the user (PDF, HTML, Word document, etc.). For example, if PDF format is specified, the server generates a PDF file using LaTeX and a PDF generation library (e.g., ReportLab). Specifically, it embeds the generated text into a LaTeX template and generates a PDF file.

[0342] Input: Generated manual content

[0343] Output: Manual converted to the specified format (e.g. PDF file)

[0344] Step 4:

[0345] Providing manuals to users

[0346] The server provides the generated manual to the user. Specifically, it generates a download link for the generated PDF file and notifies the user. The user then accesses the download link using their terminal and downloads the generated manual.

[0347] Input: Manual converted to the specified format (e.g. PDF file)

[0348] Output: The download link provided to the user, and the downloaded manual

[0349] In this way, the system generates manuals according to user requests and provides them in the specified format, making it easy to distribute and share the manuals.

[0350] (Application example 3)

[0351] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0352] Conventional manual creation systems recognize operating procedures and automatically generate screenshots, but the output format of the generated manuals is limited, making it difficult to flexibly distribute and share them according to user needs. Furthermore, there is a need to efficiently generate operation and maintenance manuals for complex equipment such as factory robots and provide them in an appropriate format. This can lead to reduced understanding of operating procedures and reduced maintenance efficiency, leading to work delays and errors.

[0353] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0354] In this invention, the server includes means for recognizing computer operation procedures, means for automatically generating screenshots and text based on the operation procedures, means for creating a manual using the generated screenshots and text, means for generating the manual using a generative AI model, and means for providing the generated manual in various output formats such as PDF, HTML, Word document, etc. This enables users to efficiently generate detailed manuals for operating and maintaining factory robots and distribute and share them in the required format.

[0355] "Means for recognizing computer operating procedures" refers to a device or software that has the function of automatically detecting and recording computer operating procedures.

[0356] "Means for automatically generating screenshots and text" refers to a device or software that has the function of automatically creating a screen image (screenshot) and its explanatory text (text) based on recognized operating procedures.

[0357] The "means for creating a manual" refers to a device or software that has the function of creating a document (manual) that summarizes operating procedures and explanations using the generated screenshots and text.

[0358] A "means for generating a manual using a generative AI model" is a device or software that has the function of automatically generating a manual based on input information using an artificial intelligence model.

[0359] "Means for providing in various output formats such as PDF, HTML, Word document, etc." refers to a device or software that has the function of outputting the generated manual in different formats such as PDF, HTML, Word document, etc., and providing it in the format selected by the user.

[0360] The system for implementing this invention has the function of recognizing computer operating procedures, automatically generating screenshots and text, and using them to create manuals.It also has the function of generating manuals using a generative AI model and providing them in various output formats such as PDF, HTML, and Word documents.

[0361] System Program

[0362] The program for this system works as follows: First, when a user inputs a computer operation procedure, the server recognizes the operation procedure. Next, screenshots and text are automatically generated based on the recognized operation procedure. The server then creates a manual using the generated screenshots and text.

[0363] Furthermore, a generative AI model is used to generate a detailed manual based on the input information. This generative AI model uses OpenAI's GPT-3 (registered trademark), etc. The generated manual is output in a format selected by the user (PDF, HTML, Word document, etc.).

[0364] Hardware and software used

[0365] The system uses the following hardware and software:

[0366] Hardware: Servers, user devices (smartphones, PCs, etc.)

[0367] Software: OpenAI API, FPDF, python-docx, BeautifulSoup

[0368] Data processing and calculation

[0369] The server analyzes the operating procedures entered by the user and generates screenshots and text. It then uses a generative AI model to generate a detailed manual. The generated text is then converted into formats such as PDF, HTML, and Word documents. Specifically, the FPDF library is used to generate PDFs, the python-docx library to generate Word documents, and the BeautifulSoup library to generate HTML.

[0370] Specific examples

[0371] For example, to generate a manual that describes the operating procedures for a factory robot, the user enters the following prompt:

[0372] Please explain the operating procedures for the factory robot, including how to power it on, how to set it up initially, basic operation, and maintenance procedures.

[0373] By inputting this prompt into a generative AI model, a manual containing detailed operating procedures is generated and output in the specified format, allowing users to efficiently obtain detailed information on the operation and maintenance of factory robots and distribute and share it in the format of their choice.

[0374] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0375] Step 1:

[0376] The user inputs the operation procedure from the terminal.

[0377] Input: The user inputs a prompt statement regarding the operating procedure of the factory robot.

[0378] Specific operation: The user enters into the terminal's input form, "Please explain the operating procedures for the factory robot. Specifically, please explain in detail how to turn it on, how to perform initial setup, basic operating procedures, and maintenance procedures."

[0379] Output: The prompt text entered is sent to the server.

[0380] Step 2:

[0381] The server recognizes the operation procedure.

[0382] Input: The prompt text sent by the user.

[0383] Specific operation: The server analyzes the received prompt text and recognizes the operation procedure.

[0384] Output: Data of recognized operation steps.

[0385] Step 3:

[0386] The server automatically generates screenshots and text.

[0387] Input: The data of the recognized operation procedure.

[0388] Specific operation: The server automatically generates relevant screenshots and their explanatory text based on the operating procedures.

[0389] Output: Generated screenshots and text.

[0390] Step 4:

[0391] The server generates the manual using the generative AI model.

[0392] Input: Generated screenshot and text.

[0393] How it works: The server uses a generative AI model (e.g., OpenAI's GPT-3) to generate a detailed manual based on screenshots and text.

[0394] Output: The generated manual text data.

[0395] Step 5:

[0396] The server provides the generated manual in formats such as PDF, HTML, Word document, etc.

[0397] Input: Generated manual text data.

[0398] Specific behavior: The server uses the FPDF library to generate PDFs, the python-docx library to generate Word documents, and the BeautifulSoup library to generate HTML.

[0399] Output: Generated manual in formats such as PDF, HTML, Word document, etc.

[0400] Step 6:

[0401] The user downloads the generated manual from the terminal.

[0402] Input: Generated manuals in formats such as PDF, HTML, Word documents, etc.

[0403] Specific operation: The user selects the desired format of the manual from the terminal and clicks the download button.

[0404] Output: The manual downloaded to the user's device.

[0405] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0406] "Example 1"

[0407] In one embodiment of the present invention, an emotion engine is incorporated. This emotion engine recognizes the user's emotions and adjusts the content of the manual based on those emotions. Specifically, if the user is having difficulty with an operation, the emotion engine recognizes that emotion and adjusts the manual to make it easier for the user to understand, such as by providing more detailed explanations for the relevant part of the manual.

[0408] "Example 2"

[0409] The emotion engine also has the ability to adjust the manual output format based on the user's emotions. For example, if the user is impatient, the emotion engine will recognize that emotion and output the manual in a concise format so that the user can quickly obtain the information they need. Conversely, if the user wants a deeper understanding, the emotion engine will recognize that emotion and output the manual in a detailed format so that the user can fully understand the information they need.

[0410] "Example 3"

[0411] Furthermore, the emotion engine adjusts the distribution and sharing method of the manual depending on the user's emotions. For example, if the user is in a hurry, the emotion engine will recognize that emotion and distribute the manual via an immediate means such as email or messaging app. Conversely, if the user wants to take their time to gather information, the emotion engine will recognize that emotion and distribute the manual via a means that allows the user to receive the information more slowly, such as by mailing a printed copy.

[0412] The processing flow of each embodiment will be described below.

[0413] "Example 1"

[0414] Step 1: The user initiates an action on the computer.

[0415] Step 2: The emotion engine recognizes the user's emotion.

[0416] Step 3: Adjust the content of the manual based on the emotions recognized by the emotion engine.

[0417] Step 4: The adjusted manual is presented to the user.

[0418] "Example 2"

[0419] Step 1: The user initiates an action on the computer.

[0420] Step 2: The emotion engine recognizes the user's emotion.

[0421] Step 3: Adjust the manual output format based on the emotions recognized by the emotion engine.

[0422] Step 4: The adjusted format of the manual is presented to the user.

[0423] "Example 3"

[0424] Step 1: The user initiates an action on the computer.

[0425] Step 2: The emotion engine recognizes the user's emotion.

[0426] Step 3: Adjust how you distribute and share the manual based on the emotions identified by the emotion engine.

[0427] Step 4: The manual is distributed to users in a coordinated manner.

[0428] Example 1

[0429] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0430] Conventional manual creation systems require manual recording of user operations, taking screenshots, and inputting text, which is extremely time-consuming. Furthermore, if a user experiences difficulty with an operation, it is difficult to adjust the manual to reflect that experience. This makes it difficult to quickly create a manual that is easy for users to understand.

[0431] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0432] In this invention, the server includes means for monitoring user operations, means for taking screenshots based on the operations, means for recording the operation contents as text, means for generating a draft of a manual using the generated screenshots and text, and means for recognizing the user's emotions and adjusting the contents of the manual based on the emotions. This makes it possible to automatically record the user's operations and quickly create an easy-to-understand manual that is adjusted according to the user's emotions.

[0433] The "means for monitoring user operations" refers to a device or software that has the function of monitoring the operations in real time when a user operates specific software.

[0434] A "means for taking a screenshot" is a device or software that has the function of taking a screenshot of the screen corresponding to a user operation each time that operation is recognized.

[0435] The "means for recording operation content as text" refers to a device or software that has the function of recording operations performed by a user in text format.

[0436] A "means for generating a manual draft" is a device or software that has the function of automatically creating a draft of a manual that explains operating procedures using the generated screenshots and text.

[0437] The "means for recognizing user's emotions" refers to a device or software that has the function of analyzing the emotions felt by the user in response to an operation and recognizing those emotions.

[0438] The "means for adjusting the contents of the manual" refers to a device or software that has the function of adjusting the explanation contents of the manual based on the recognized user's emotions to make them easier for the user to understand.

[0439] This invention is a manual creation assistant system that recognizes operations performed by a user on a computer and automatically generates screenshots and text corresponding to those operations. A specific embodiment of this system will be described below.

[0440] First, the device monitors user operations. Specifically, operation recognition software (e.g., OBS Studio or Snagit) installed on the device captures user operations in real time. When a user operates a specific software, the device recognizes the operation and takes a corresponding screenshot.

[0441] Next, the device records the user's operation as text. For example, if the user selects "Open" from the "File" menu to open an image file, the device generates text such as "Select Open from the File menu to open an image file."

[0442] The server then generates a draft manual based on the screenshots and text sent from the device. The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate instructions for the operation procedures. For example, for the operation "Launch image processing software and open an image file," the server generates an explanation such as "Launch the image processing software. Next, select 'File' -> 'Open' from the menu and open the image file you want to edit."

[0443] Furthermore, the server uses an emotion engine (e.g., Affectiva or IBM Watson® Tone Analyzer) to analyze the user's emotions. If the user is having difficulty with an operation, the emotion engine recognizes the emotion and the server provides more detailed explanations. For example, if the user is having trouble using the brush tool, the server adds a detailed explanation such as, "Select the brush tool. You can adjust the brush size and opacity to fill in more detailed areas."

[0444] As a concrete example, consider the case where a user edits an image using image processing software. When the user starts the image processing software and opens an image file, the device recognizes this, takes a screenshot, and records the operation as text. The server uses this data to generate a draft manual like the one below:

[0445] 1. Start the image processing software (Screenshot 1).

[0446] 2. Select "File" → "Open" from the menu and open the image file you want to edit (Screenshot 2).

[0447] 3. Click the "Create New Layer" button in the "Layers" panel at the bottom right (Screenshot 3).

[0448] 4. Select the Brush Tool from the left toolbar (Screenshot 4).

[0449] 5. Click and drag over the part of the image you want to fill (Screenshot 5).

[0450] If the emotion engine recognizes the user's confusion, the server refines the explanation as follows:

[0451] 5. Click and drag over the area of ​​the image you want to fill. You can adjust the brush size and opacity to fill in more detail. You can change the brush settings in the options bar at the top (Screenshot 6).

[0452] An example of a prompt to input to a generative AI model is as follows:

[0453] You are creating a manual that will guide users through the steps of editing images using image processing software. Please generate instructions for each step based on the following steps:

[0454] 1. Launch the image processing software

[0455] 2. Open the image file

[0456] 3. Add layers

[0457] 4. Select the Brush Tool

[0458] 5. Fill in part of an image

[0459] In this way, a manual is automatically generated based on the user's operations.

[0460] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0461] Step 1:

[0462] The user operates a specific piece of software.

[0463] Input: User actions

[0464] Output: Operation event

[0465] Specific operation: The user starts the image processing software and opens an image file.

[0466] Step 2:

[0467] The terminal monitors the user's operations.

[0468] Input: Operation event

[0469] Output: Operation log

[0470] Specific operation: Operation recognition software (e.g., OBS Studio or Snagit) installed on the device captures user operations in real time and generates an operation log.

[0471] Step 3:

[0472] The device will take a screenshot based on your actions.

[0473] Input: Operation log

[0474] Output: Screenshot

[0475] Specific behavior: The moment a user opens an image in image processing software, the device takes a screenshot of that screen.

[0476] Step 4:

[0477] The device records the operations as text.

[0478] Input: Operation log

[0479] Output: Operation text

[0480] Specific operation: When a user selects "Open" from the "File" menu to open an image file, the device generates text such as "Select Open from the File menu to open an image file."

[0481] Step 5:

[0482] The server generates a draft of the manual using the generated screenshots and text.

[0483] Input: Screenshot, Operation text

[0484] Output: Manual Draft

[0485] Specific operation: The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate instructions for the operation. For example, for the operation "Launch image processing software and open an image file," the server generates an instruction such as "Launch the image processing software. Next, select 'File' -> 'Open' from the menu and open the image file you want to edit."

[0486] Step 6:

[0487] The server recognizes the user's emotions and adjusts the content of the manual accordingly.

[0488] Input: User operation log, operation text

[0489] Output: Adjusted manual draft

[0490] Specific operation: The server uses an emotion engine (e.g., Affectiva or IBM Watson Tone Analyzer) to analyze the user's emotions. If the user is having difficulty with an operation, the emotion engine recognizes that emotion and the server provides more detailed explanations. For example, if the user is having trouble using the brush tool, the server adds a detailed explanation such as, "Select the brush tool. You can adjust the brush size and opacity to fill in finer details."

[0491] (Application example 1)

[0492] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0493] Conventional manual creation systems require users to record operating procedures and manually generate screenshots and text, which is time-consuming and labor-intensive. Furthermore, they provide uniform explanations without considering the user's feelings, which means they are unable to provide sufficient support to users who have difficulty operating the system. Operating factory robots, in particular, requires complex procedures and specialized knowledge, which often confuses users. This increases the risk of operating errors and reduced efficiency.

[0494] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0495] In this invention, the server includes means for recognizing computer operation procedures, means for automatically generating screenshots and text based on the operation procedures, means for creating a manual using the generated screenshots and text, means for recognizing a user's emotion, and means for adjusting the content of the manual based on the recognized emotion. This automatically records the user's operation procedures and provides appropriate explanations according to the emotion, thereby reducing operation errors and enabling efficient manual creation.

[0496] "Means for recognizing computer operation procedures" refers to technology that automatically detects the operations a user performs on a computer and records those procedures.

[0497] "Means for automatically generating screenshots and text" is a technology that automatically creates screen capture images and their explanatory text based on recognized operating procedures.

[0498] The "means for creating a manual" is a technology that automatically creates a manual in a format that can be referenced by users using the generated screenshots and text.

[0499] "Means for recognizing user emotions" refers to technology that detects emotions from a user's facial expressions and behavior and determines their emotional state.

[0500] "Means for adjusting the content of the manual based on recognized emotions" refers to a technology that dynamically changes the explanation content and level of detail of the manual depending on the user's emotional state.

[0501] A system for carrying out the present invention has the following configuration: a server includes means for recognizing computer operating procedures, means for automatically generating screenshots and text, means for creating a manual using the generated screenshots and text, means for recognizing a user's emotion, and means for adjusting the content of the manual based on the recognized emotion.

[0502] Hardware and software used

[0503] Hardware: Computer, camera (for emotion recognition)

[0504] Software: Python, OpenCV (for image processing), PyAutoGUI (for taking screenshots), EmotionRecognizer (a custom library for emotion recognition), TextGenerator (a custom library for text generation)

[0505] Processing flow

[0506] The server first recognizes the user's computer operation procedures. Specifically, it detects and records the user's operation procedures in real time when the user operates specific software. Next, it automatically generates screenshots and text based on the recognized operation procedures. Screenshots are taken using PyAutoGUI, and text is generated using TextGenerator.

[0507] The server then captures the user's facial expressions using a camera and recognizes their emotions using EmotionRecognizer. Based on the recognized emotions, the server adjusts the content of the generated manual. For example, if the user is confused, it adds detailed explanations to the text.

[0508] Specific examples

[0509] For example, when recording the operating procedures of a factory robot, the server recognizes the robot startup procedure, part installation procedure, and operation check procedure, and generates screenshots and text corresponding to each step. If the user is confused, adding detailed explanations can reduce operating errors.

[0510] Prompt Sentence Examples

[0511] "Write a program that records the robot's startup steps and adds detailed instructions if the operator is confused."

[0512] In this way, by automatically recording the user's operating procedures and providing appropriate explanations according to their emotions, it is possible to reduce operating errors and create manuals efficiently.

[0513] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0514] Step 1:

[0515] The server recognizes the user's computer operation procedures. Specifically, when the user operates specific software, the operation procedures are detected and recorded in real time. The input is the user's operation, and the output is the data of the recognized operation procedures. This data is used in subsequent processing.

[0516] Step 2:

[0517] The server automatically generates screenshots based on the recognized operation steps. Specifically, it uses PyAutoGUI to capture images of the screen corresponding to the operation steps. The input is the data of the operation steps recognized in step 1, and the output is the captured screenshot image.

[0518] Step 3:

[0519] The server automatically generates text based on the recognized operation procedure. Specifically, it uses TextGenerator to generate an explanation of the operation procedure. The input is the data of the operation procedure recognized in step 1, and the output is the generated explanation text.

[0520] Step 4:

[0521] The server captures the user's facial expressions using a camera and recognizes the user's emotions using EmotionRecognizer. The input is the user's facial expression image, and the output is the recognized emotion data. This emotion data is used in subsequent processing.

[0522] Step 5:

[0523] The server adjusts the content of the manual based on the recognized emotions. Specifically, if the user is confused, it adds a detailed explanation to the generated explanatory text. The input is the explanatory text generated in step 3 and the emotion data recognized in step 4, and the output is the adjusted explanatory text.

[0524] Step 6:

[0525] The server creates a manual using the generated screenshots and adjusted explanatory text. The input is the screenshot images obtained in step 2 and the explanatory text adjusted in step 5, and the output is the completed manual. This manual is provided in a format that can be referenced by the user.

[0526] Example 2

[0527] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0528] In conventional manual creation systems, recording operation procedures, generating screenshots, and collecting text are often done manually, resulting in low work efficiency. Furthermore, the manual output format cannot be adjusted based on the user's emotions, making it difficult to provide information that meets the user's needs. Furthermore, the lack of an automatic table of contents or index generation function makes the manual difficult to read, making it difficult to quickly find the information needed. To solve these issues, a system is needed that can automatically recognize operation procedures, automatically generate screenshots and text, adjust the output format based on the user's emotions, and automatically generate a table of contents and index.

[0529] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0530] In this invention, the server includes means for recognizing computer operating procedures, means for automatically generating screenshots and text based on the operating procedures, means for creating a manual using the generated screenshots and text, and means for analyzing a user's emotions and adjusting the output format of the manual based on the analysis results. This enables automatic recognition of operating procedures and automatic generation of screenshots and text, thereby realizing appropriate manual output according to the user's emotions. In addition, the automatic generation function of a table of contents and index improves the readability of the manual and enables users to quickly find necessary information.

[0531] The "means for recognizing computer operation procedures" is a function for detecting operations performed by a user on a computer and recording those procedures.

[0532] A "screenshot" is a technique for capturing an image of what is displayed on a computer screen.

[0533] "Text" refers to text information entered by the user or explanatory text automatically generated by the system.

[0534] The "means for creating a manual" is a function that combines the generated screenshots and text to automatically generate a manual in a format that is easy for the user to understand.

[0535] "Means for analyzing user emotions" refers to technology that infers emotions from the user's facial expressions, behavior, input content, etc., and performs analysis based on that information.

[0536] The "means for adjusting the output format of the manual based on the analysis results" is a function for changing the content and format of the manual according to the results of the user's emotion analysis.

[0537] "Function to automatically generate a table of contents or index" is a technology that automatically creates a table of contents or index based on the contents of a manual.

[0538] This invention is a system that recognizes the operating procedures performed by a user on a computer, automatically generates screenshots and text based on those procedures, and creates a manual using the generated screenshots and text.It also has the function of analyzing the user's emotions and adjusting the output format of the manual based on the analysis results.

[0539] Hardware and software used

[0540] This system uses the following hardware and software:

[0541] Terminal: A device such as a computer or smartphone that a user operates.

[0542] Server: A server for organizing data, creating manuals, and performing sentiment analysis.

[0543] Emotion engine: Software for analyzing user emotions.

[0544] Manual creation software: Software for creating manuals, such as LaTeX or Microsoft Word.

[0545] Data processing and calculation

[0546] 1. User operation records

[0547] When a user operates the software, the device records the operation in real time, specifically saving the buttons the user clicks and the contents of text fields they enter as logs.

[0548] 2. Generate screenshots and text

[0549] The device will automatically generate screenshots based on user actions, for example, capturing the screen at the moment the user clicks the "Install" button.

[0550] At the same time, the device collects the text entered by the user and the explanation automatically generated by the system, for example, "Next, click the Install button."

[0551] 3. Sending and organizing data

[0552] The device sends the generated screenshot and text to the server.

[0553] The server then organizes the data it receives, associating screenshots with corresponding text and arranging them in chronological order.

[0554] 4. Creating a manual

[0555] The server creates manuals based on the organized data, using software such as LaTeX and Microsoft Word.

[0556] The server will place the screenshot and text in the appropriate location and format it for easy viewing, for example by placing a corresponding description below the screenshot.

[0557] 5. Automatic table of contents and index generation

[0558] The server automatically generates a table of contents and index based on the contents of the manual, allowing users to easily find the information they need.

[0559] For example, add an "Installation Instructions" section to the table of contents and automatically insert its page number.

[0560] 6. Adjusting output format using emotion engine

[0561] When a user browses the manual, the device analyzes the user's emotions using an emotion engine.

[0562] For example, if the user is impatient, the terminal recognizes this emotion and instructs the server to output a manual in a concise format.

[0563] Conversely, if the user feels that he or she wants to understand more deeply, the terminal instructs the server to output the manual in a detailed format.

[0564] Specific examples

[0565] For example, consider a case where a user wants to learn how to install new software. As the user proceeds with the installation, the device records the operation as screenshots, and simultaneously collects comments entered by the user and explanations automatically generated by the system. This data is sent to a server, which then organizes it and creates a manual. Furthermore, if the user is impatient, the emotion engine outputs a concise manual, or if the user wants a deeper understanding, it outputs a detailed manual.

[0566] Prompt Sentence Examples

[0567] Examples of prompts to be input to a generative AI model include:

[0568] "Write a manual that explains the installation process for new software. Be concise if users are impatient, or detailed if they want a deeper understanding."

[0569] By using this prompt sentence, the generative AI model can generate an appropriate manual that corresponds to the user's emotions.

[0570] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0571] Step 1:

[0572] User operation records

[0573] Input: What you do on the computer (clicks, typing text, etc.)

[0574] How it works: When a user interacts with the software, the device records that interaction in real time, including saving the buttons the user clicks and the contents of text fields they enter as logs.

[0575] Output: Operation log data

[0576] Step 2:

[0577] Screenshot and text generation

[0578] Input: Operation log data

[0579] How it works: The device automatically generates screenshots based on user actions. For example, it captures the screen at the moment the user clicks the "Install" button. At the same time, the device collects text entered by the user and an explanation automatically generated by the system. For example, it generates an explanation such as "Next, click the Install button."

[0580] Output: Screenshot image and text data

[0581] Step 3:

[0582] Sending and organizing data

[0583] Input: Screenshot image and text data

[0584] How it works: The device sends the generated screenshots and text to the server, which then organizes the data it receives, associating screenshots with their corresponding text and arranging them in chronological order.

[0585] Output: Organized screenshots and text data

[0586] Step 4:

[0587] Manual creation

[0588] Input: Organized screenshots and text data

[0589] How it works: The server creates a manual based on the organized data. This can be done using software such as LaTeX or Microsoft Word. The server places screenshots and text in the appropriate locations and formats them in a readable format. For example, it places corresponding explanatory text below each screenshot.

[0590] Output: Manual data

[0591] Step 5:

[0592] Automatic table of contents and index generation

[0593] Input: Manual Data

[0594] What it does: The server automatically generates a table of contents and index based on the contents of the manual, allowing users to easily find the information they need. For example, it adds an "Installation Instructions" section to the table of contents and automatically inserts its page number.

[0595] Output: Manual data with table of contents and index added

[0596] Step 6:

[0597] Adjusting output format with emotion engine

[0598] Input: User emotion data, manual data

[0599] How it works: When a user is viewing a manual, the device analyzes the user's emotions using an emotion engine. For example, if the user is anxious, the device will recognize this emotion and instruct the server to output the manual in a concise format. Conversely, if the user wants to understand more deeply, the device will instruct the server to output the manual in a detailed format.

[0600] Output: Manual data in a format that corresponds to the user's emotions

[0601] (Application example 2)

[0602] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0603] Conventional manual creation systems recognize operating procedures and automatically generate screenshots and text, but they are unable to adjust the output format based on the user's emotions, making it difficult to provide the information users need quickly and appropriately. Furthermore, creating manuals for operating and configuring factory robots requires flexible responses based on the operator's emotions, but conventional systems are inadequate in this regard.

[0604] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0605] In this invention, the server includes means for recognizing computer operation procedures, means for automatically generating screenshots and text based on the operation procedures, means for creating a manual using the generated screenshots and text, means for analyzing a user's emotions, and means for adjusting the output format of the manual based on the analyzed emotions, thereby making it possible to provide the manual in a format appropriate to the user's emotions.

[0606] The "means for recognizing computer operation procedures" is a function for detecting a series of operations executed on a computer and recording the procedures.

[0607] "Means for automatically generating screenshots and text" is a function for automatically creating screen images (screenshots) and corresponding explanatory text (text) based on operating procedures.

[0608] The "means for creating a manual" is a function for combining the generated screenshots and text to create a manual in a format that can be referenced by the user.

[0609] "Means for analyzing user emotions" refers to a function that infers emotions from the user's input and behavior and analyzes those emotions as data.

[0610] "Means for adjusting the output format of the manual based on emotions" is a function that changes the content and format of the manual according to the analyzed emotions of the user, and provides information to the user in the most optimal form.

[0611] The "function to automatically generate a table of contents and index" is a function that automatically creates a table of contents and index based on the contents of the manual, allowing users to quickly find the information they need.

[0612] The "function to provide in various output formats" refers to the function to output the generated manual in multiple formats, such as PDF, HTML, and text file, and provide it in a form that is easy for users to use.

[0613] A system for carrying out the present invention has the following configuration: a server includes means for recognizing computer operating procedures, means for automatically generating screenshots and text, means for creating a manual using the generated screenshots and text, means for analyzing a user's emotions, and means for adjusting the output format of the manual based on the analyzed emotions.

[0614] Program processing explanation

[0615] The server first recognizes the computer's operating procedures. This is done by monitoring operation logs and user input to detect specific operating procedures. Next, it automatically generates screenshots and text based on the recognized operating procedures. The screenshots are generated using a camera or screen capture software (e.g., OpenCV) to capture the screen during operation, and the text is generated using OCR technology (e.g., pytesseract) to explain the operation.

[0616] The generated screenshots and text are then combined by the manual creation tool to create a user-readable manual, which is then output in PDF format using a PDF generation library such as FPDF.

[0617] Furthermore, the server uses a sentiment analysis model (e.g., the sentiment-analysis pipeline in the transformers library) to analyze the user's emotions. It infers emotions from the user's input and behavior, and adjusts the output format of the manual based on those emotions. For example, if the user is impatient, it generates a concise manual, but if the user wants more information, it generates a detailed manual.

[0618] Specific examples

[0619] For example, if an operator is impatient while operating a factory robot, the emotion engine will recognize that emotion and generate a concise manual. Conversely, if the operator wants more detailed information, the emotion engine will recognize that emotion and generate a detailed manual.

[0620] Prompt Sentence Examples

[0621] "Generate an operating manual for a factory robot. Use screenshots and operation logs to adjust output based on emotions."

[0622] In this way, it is possible to provide a manual in a format appropriate to the user's feelings.

[0623] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0624] Step 1:

[0625] The server recognizes the operating procedures of the computer.

[0626] Input: User operation logs and input data.

[0627] Processing: Analyze the operation log and detect specific operation steps.

[0628] Output: A list of recognized operation steps.

[0629] Step 2:

[0630] The server automatically generates screenshots and text based on the recognized operating procedures.

[0631] Input: A list of recognized operating steps.

[0632] Processing: Use screen capture software (e.g., OpenCV) to capture the screen during operation, and use OCR technology (e.g., pytesseract) to generate text describing the operation.

[0633] Output: Screenshot image and corresponding text.

[0634] Step 3:

[0635] The server creates a manual using the generated screenshots and text.

[0636] Input: Screenshot image and corresponding text.

[0637] Processing: Use a PDF generation library such as FPDF to create a PDF manual that combines screenshots and text.

[0638] Output: Manual in PDF format.

[0639] Step 4:

[0640] The server analyzes the user's emotions.

[0641] Input: User input data and behavior logs.

[0642] Processing: Analyze user sentiment using a sentiment analysis model (e.g., the sentiment-analysis pipeline in the transformers library).

[0643] Output: User sentiment data.

[0644] Step 5:

[0645] The server adjusts the manual output format based on the analyzed sentiment.

[0646] Input: User emotion data and a PDF manual.

[0647] Processing: Change the content and format of the manual based on the emotion data. For example, generate a concise manual if the user is impatient, or a detailed manual if the user wants more information.

[0648] Output: Manual in adjusted format.

[0649] Step 6:

[0650] The server provides the final manual to the user.

[0651] Input: Manual in adjusted format.

[0652] Action: Distribute the manual to users and share it as needed.

[0653] Output: User-referenceable manual.

[0654] Example 3

[0655] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0656] Conventional manual creation systems have the problem that it is difficult to provide information that meets the needs of users because they do not provide a flexible distribution method that responds to the user's emotions and situation.In addition, it is difficult to distribute and share the generated manuals, making it difficult to communicate information efficiently.

[0657] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means. In this invention, the server includes a means for recognizing computer operation procedures, a means for automatically generating images and text, a means for creating a document using the generated images and text, a means for analyzing a user's emotions, a means for determining a document distribution method based on the analysis result, and a means for distributing the document in the determined method. This enables a flexible distribution method according to the user's emotions and situation, thereby realizing efficient information transmission.

[0658] "Means for recognizing computer operating procedures" refers to a device or software for monitoring computer operations and recording those operating procedures.

[0659] "Means for automatically generating images and text" means a device or software for automatically generating associated images and text based on recognized operating procedures.

[0660] A "means for creating a document" is a device or software that uses the generated images and text to create a document in a format that is usable by a user.

[0661] "Means for analyzing user emotions" refers to a device or software for inferring and analyzing emotions from user input and behavior.

[0662] The "means for determining a document distribution method based on the analysis results" refers to a device or software for selecting the optimal document distribution method based on the results of the user emotion analysis.

[0663] A "means for distributing a document in a determined manner" is a device or software for providing a document to a user in accordance with a selected distribution method.

[0664] This invention is a system that recognizes user operation procedures, automatically generates images and text to create documents, and analyzes user emotions to determine the optimal distribution method. Specific embodiments of this system are described below.

[0665] First, the user enters data about the contents of the manual. This data may come in the form of text files, images, tables, etc. For example, the user may upload a text file about how to use a new piece of software.

[0666] The server then receives the data entered by the user and stores it in the server's storage, ready for analysis. For example, the server stores the uploaded text file in a specific directory.

[0667] The server then analyzes the received data using natural language processing (NLP) techniques to understand the content of the data. For example, the server might use a Python NLP library (e.g., spaCy) to analyze the content of the text file and extract important keywords and phrases.

[0668] The server then generates a document based on the analysis results, outputting the document in the format specified by the user (PDF, HTML, Word document, etc.) For example, the server may generate a PDF document using LaTeX.

[0669] Furthermore, the server analyzes the user's emotions using an emotion engine, which infers emotions from the user's input and behavior. For example, the server uses TextBlob to analyze emotions such as "I'm in a hurry" or "I want to take my time to get information" from the user's input text.

[0670] The server then decides how to distribute the document based on the results of the emotion engine. For example, if the user is in a hurry, the server may decide to distribute the document via email or a messaging app.

[0671] Finally, the server distributes the document in the manner determined: for example, the server may send the generated PDF document as an email attachment to the user, or, if the user prefers to dig deeper, print the document and mail it.

[0672] Specific examples

[0673] Let's say a user needs a manual on how to use new software, and they want the information quickly.

[0674] 1. The user uploads a text file to the server with instructions on how to use the new software.

[0675] 2. The server saves the uploaded text file in a specific directory.

[0676] 3. The server uses a Python NLP library (e.g., spaCy) to parse the contents of the text file and extract important keywords and phrases.

[0677] 4. The server generates a PDF manual using LaTeX.

[0678] 5. The server uses TextBlob to parse the "hurry" sentiment from the user's input text.

[0679] 6. The server realizes that the user is in a hurry and decides to distribute the manual by email.

[0680] 7. The server attaches the generated PDF manual to an email and sends it to the user.

[0681] Prompt Sentence Examples

[0682] "Generate a manual in PDF format on how to use new software and email it to users in a hurry."

[0683] In this way, the server can generate documents that meet the needs of the user and distribute them in an appropriate manner. The flow of the specific processing in the third embodiment will be described with reference to FIG.

[0684] Step 1:

[0685] The user inputs data. The user uploads data such as text files, images, and tables related to the contents of the manual from their terminal to the server. The input data is, for example, a text file on how to use new software. The output is the input data saved on the server.

[0686] Step 2:

[0687] The server receives the data. The server saves the data uploaded by the user in a specific directory. The input is the text file uploaded by the user, and the output is a file saved in the server's storage. Specifically, the server checks the directory where the file is saved and saves the file.

[0688] Step 3:

[0689] The server analyzes the data. The server uses a Python NLP library (e.g., spaCy) to analyze the contents of the saved text file. The input is the text file saved on the server, and the output is the analysis results: important keywords and phrases. Specifically, the server reads the text file and analyzes the text using the NLP library.

[0690] Step 4:

[0691] The server generates a document. The server generates a document based on the analysis results. It outputs the document in the format specified by the user (PDF, HTML, Word document, etc.). The input is keywords or phrases resulting from the analysis, and the output is the generated document. Specifically, the server uses LaTeX to generate a PDF document.

[0692] Step 5:

[0693] The server applies the emotion engine. The server uses the emotion engine to analyze the user's emotion. The input is the user's input text and behavioral data, and the output is the analyzed emotion information. Specifically, the server uses TextBlob to analyze the emotion from the user's input text.

[0694] Step 6:

[0695] The server determines the distribution method. The server determines the distribution method for the document based on the results of the emotion engine. The input is the analyzed emotion information, and the output is the determined distribution method. In concrete terms, the server analyzes the emotion information and selects the optimal distribution method.

[0696] Step 7:

[0697] The server distributes the document. The server provides the document to the user in the determined manner. The input is the generated document and the determined distribution method, and the output is the document distributed to the user. Specifically, the server attaches the generated PDF document to the user's email and sends it. If necessary, the server also prints and mails the document.

[0698] (Application example 3)

[0699] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0700] Conventional manual generation systems lack the flexibility to respond to user emotions and situations, making it difficult to provide information in an appropriate format to users who want to learn at their own pace, while being unable to provide information in an immediate manner to users who are in a hurry.In addition, the distribution methods for generated manuals are limited, making it difficult to provide them in a variety of formats to meet user needs.

[0701] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0702] In this invention, the server includes means for recognizing computer operation procedures, means for automatically generating screenshots and text, means for creating a manual using the generated screenshots and text, means for recognizing a user's emotion, means for adjusting the method for distributing the manual based on the recognized emotion, and means for providing the generated manual in various output formats. This makes it possible to provide a manual in an optimal format according to the user's emotion and situation, making it possible to provide information in an immediate manner to users who are in a hurry and information in an appropriate format to users who want to learn at their own pace.

[0703] "Means for recognizing computer operating procedures" refers to a device or software that monitors computer operations and analyzes and recognizes those operating procedures.

[0704] "Means for automatically generating screenshots and text" refers to a device or software for automatically generating a captured image of a screen (screenshot) and corresponding text based on recognized operating procedures.

[0705] The "means for creating a manual" refers to a device or software that uses the generated screenshots and text to create a manual in a format that is easy for users to understand.

[0706] A "means for recognizing a user's emotions" is a device or software that analyzes a user's facial expressions, voice, actions, etc. to recognize their emotions.

[0707] The "means for adjusting the method of distributing the manual based on the recognized emotion" is a device or software for optimizing the method of distributing the manual according to the emotion of the user.

[0708] "Means for providing the generated manual in various output formats" refers to a device or software for outputting the generated manual in different formats, such as PDF, HTML, or Word document, and providing it to the user.

[0709] An embodiment of the present invention will be described below. The following system is an emotion-responsive manual generation system installed on a factory robot.

[0710] The server first uses a means for recognizing computer operation procedures to monitor user operations and analyze and recognize those operation procedures. Next, it uses a means for automatically generating screenshots and text to automatically generate screen capture images (screenshots) and corresponding text based on the recognized operation procedures. The generated screenshots and text are compiled into a manual in a format that is easy for users to understand using a means for creating a manual.

[0711] Furthermore, a means for recognizing user emotions is used to analyze the user's facial expressions, voice, actions, etc. to recognize their emotions. Based on the recognized emotions, a means for adjusting the method of distributing the manual is used to select the optimal distribution method according to the user's emotions. For example, if the user is in a hurry, the manual can be distributed via a highly immediate method such as email or a messaging app, but if the user wants to take their time to understand the information, a method such as mailing a printed manual can be used.

[0712] The generated manuals can be output in various formats, such as PDF, HTML, and Word documents, and provided to users, making it possible to provide information in a variety of formats according to user needs.

[0713] Specific hardware and software include the Emotion Recognition API (e.g., Microsoft Azure (registered trademark) Emotion API) for the emotion engine, the Manual Generator Module (e.g., a custom module created in Python) for manual generation, and the Twilio API for SMS and SMTP for Email for the communication module.

[0714] For example, if the emotion engine determines that a worker is in a hurry, a PDF manual will be sent to the worker's smartphone instantly. If the emotion engine determines that the worker wants to take their time to learn, a printed manual will be sent by mail.

[0715] Examples of prompts for generative AI models include:

[0716] "If users are in a hurry, distribute the manual in a timely manner. For example, send a PDF manual by email. If users want to study at their own pace, mail a printed manual."

[0717] The above is an embodiment of the present invention.

[0718] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0719] Step 1:

[0720] The server uses a means for recognizing computer operation procedures to monitor user operations and analyze and recognize those operation procedures. Inputs include the user's operation log and interface event data. The output is a list of recognized operation procedures. In this step, the operation log is analyzed and event data is collected.

[0721] Step 2:

[0722] The server uses a means for automatically generating screenshots and text to automatically generate screen capture images (screenshots) and corresponding text based on the recognized operating procedures. The input includes the list of operating procedures obtained in step 1. The output is pairs of screenshot images and text. In this step, screen captures are obtained and text is generated.

[0723] Step 3:

[0724] The server uses the generated screenshots and text to create a user-friendly manual. The input includes the screenshot image and text pairs obtained in step 2. The output is the completed manual. In this step, the screenshots and text are integrated and the manual is formatted.

[0725] Step 4:

[0726] The server uses a means to recognize the user's emotions by analyzing the user's facial expressions, voice, and behavior. The input includes the user's facial expression data and voice data. The output is a label for the recognized emotion. In this step, the emotion recognition algorithm is applied and the data is analyzed.

[0727] Step 5:

[0728] The server uses a means to adjust the manual distribution method based on the recognized emotions. The input includes the emotion labels obtained in step 4. The output is the determination of the optimal distribution method. In this step, the selection of the distribution method is based on the emotion labels.

[0729] Step 6:

[0730] The server uses a means to provide the generated manual in various output formats, outputting it in different formats such as PDF, HTML, Word document, etc., and provides it to the user. The input includes the completed manual obtained in step 3 and the distribution method determined in step 5. The output is the manual provided to the user. In this step, the format of the manual is converted and distributed.

[0731] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0732] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0733] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0734] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0735] [Second embodiment]

[0736] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0737] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0738] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0739] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0740] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0741] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0742] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0743] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0744] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0745] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0746] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0747] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0748] "Example 1"

[0749] The present invention is a manual creation assistant system. This system recognizes the operations a user performs on a computer and automatically generates screenshots and text corresponding to those operations. Specifically, when a user operates a specific piece of software, the system recognizes the operating procedure and generates corresponding screenshots and explanatory text. This information forms the basis for creating a manual.

[0750] "Example 2"

[0751] Furthermore, this embodiment of the present invention has a function to automatically create a manual using the generated screenshots and text. Specifically, the generated screenshots and text are combined, organized, and compiled into a manual format. It also has a function to automatically generate a table of contents and index for the manual. This makes it possible to significantly reduce the effort required to create a manual.

[0752] "Example 3"

[0753] Furthermore, the present invention has the function of providing the generated manual in various output formats. Specifically, the manual can be output in a format required by the user, such as PDF, HTML, or Word document. This makes it easy to distribute and share the manual.

[0754] The processing flow of each embodiment will be described below.

[0755] "Example 1"

[0756] Step 1: The user operates a specific piece of software.

[0757] Step 2: The system recognizes the user's operating procedure.

[0758] Step 3: The system will generate screenshots and explanatory text corresponding to the operating steps.

[0759] "Example 2"

[0760] Step 1: Get the screenshots and text generated by the system.

[0761] Step 2: The system combines this information and compiles it into a manual format.

[0762] Step 3: The system automatically generates a table of contents and index for the manual.

[0763] "Example 3"

[0764] Step 1: Obtain the system-generated manual.

[0765] Step 2: The system outputs the manual in the format required by the user, such as PDF, HTML, Word document, etc.

[0766] Example 1

[0767] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0768] Traditional manual creation required manually recording operation procedures, taking screenshots, and creating explanatory text, which was extremely time-consuming and labor-intensive. Furthermore, there was a high possibility of human error when recording operation procedures and creating explanatory text. Furthermore, creating a table of contents and index for the generated manual, providing it in various output formats, and distributing and sharing it were also time-consuming and labor-intensive.

[0769] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0770] In this invention, the server includes a means for monitoring user operations, a means for recording the operation details, a means for taking screenshots in response to the operations, a means for generating explanatory text using a generative AI model, and a means for creating a manual using the generated screenshots and explanatory text. This allows for real-time monitoring of user operations, automatic generation of screenshots and explanatory text based on the operation procedures, and efficient and accurate creation of manuals. It also enables automatic generation of a table of contents and index for the manual, provision in various output formats, and easy distribution and sharing.

[0771] A "means for monitoring user operations" is a device or software that has the function of monitoring the operations performed by a user on a computer in real time and capturing operation events.

[0772] "Means for recording operation details" refers to a device or software that has the function of recording the operation details of the monitored user in a log file or database and saving them in a format that can be referenced later.

[0773] A "means for taking a screenshot" is a device or software that has the function of capturing an image of the screen at an appropriate time in response to a user's operation and saving it as an image file.

[0774] A "means for generating explanatory text using a generative AI model" is a device or software that has the function of automatically generating explanatory text in natural language using a generative AI model based on recorded operation content.

[0775] A "means for creating a manual" is a device or software that has the function of combining the generated screenshots with explanatory text to form each page of the manual and to create a complete manual including a table of contents and index.

[0776] The "function to automatically generate a table of contents and index" is a function that automatically generates a table of contents and index based on the contents of the manual, allowing users to easily search for the information they need.

[0777] "Ability to provide in various output formats" refers to the ability to output the generated manual in different file formats such as PDF, HTML, DOCX, etc., and provide it flexibly according to the user's needs.

[0778] "Functions that make distribution and sharing easy" refers to functions that allow the generated manuals to be easily distributed and shared via email, cloud storage, an internal network, etc.

[0779] MODE FOR CARRYING OUT THE INVENTION

[0780] This invention is a manual creation assistant system that recognizes user operations on a computer and automatically generates screenshots and explanatory text corresponding to those operations. This system monitors user operations in real time and automatically generates screenshots and explanatory text based on the operating procedures, enabling efficient and accurate manual creation.

[0781] Hardware and software used

[0782] 1. Terminal

[0783] A computer device that monitors and records user operations.

[0784] Specifically, it includes software that captures events such as user mouse clicks, keyboard input, and window switching.

[0785] 2. Server

[0786] A computer device that generates explanatory text based on recorded operations and screenshots to create manuals.

[0787] Use a generative AI model (e.g., OpenAI's GPT-4) to create text that explains operating procedures in natural language.

[0788] Data processing and calculation

[0789] 1. Monitor user actions

[0790] The terminal monitors the user's operations in real time when using specific software.

[0791] Specifically, the terminal captures events such as user mouse clicks, keyboard input, and window switches.

[0792] 2. Record your actions

[0793] The terminal records the monitored operation details in a log file.

[0794] Specifically, it includes the type of operation, a timestamp, and information about the window or application that is the target of the operation.

[0795] 3. Take a screenshot

[0796] The device will automatically take screenshots at appropriate times depending on the user's actions.

[0797] Specifically, a screenshot is captured immediately after the operation is completed and saved as an image file.

[0798] 4. Generate explanatory text

[0799] The server generates explanatory text based on the recorded operations and screenshots.

[0800] Use a generative AI model (e.g., OpenAI's GPT-4) to create text that explains operating procedures in natural language.

[0801] 5. Configuring the manual

[0802] The server combines the acquired screenshots with the generated explanatory text to create each page of the manual.

[0803] Specifically, screenshots and corresponding explanatory text are laid out in HTML or PDF format and saved as a manual.

[0804] Specific examples

[0805] The user performs image editing operations using image processing software.

[0806] The device monitors and records when a user opens the "Filter" menu and selects the "Blur" option.

[0807] Your device will take a screenshot of the moment you select the "Blur" option.

[0808] Based on the recorded actions and screenshots, the server generates the following explanatory text: "Open the 'Filter' menu and select the 'Blur' option."

[0809] The server combines screenshots and explanatory text to form one page of the manual.

[0810] Prompt Sentence Examples

[0811] "Please explain the steps a user would take to edit an image in your image processing software, including applying a filter."

[0812] In this way, manuals can be created efficiently by monitoring user operations and automatically generating screenshots and explanatory text.

[0813] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0814] Step 1:

[0815] Monitor user actions

[0816] The terminal monitors the user's operations in real time when using specific software.

[0817] Input: User interaction events such as mouse clicks, keyboard input, and window switching.

[0818] Data processing: Capture operation events and extract information about the operation type, timestamp, and the window or application being operated on.

[0819] Output: The captured operation event data.

[0820] Specific behavior: Monitors when a user clicks the "New" button in Microsoft Word.

[0821] Step 2:

[0822] Record your operations

[0823] The terminal records the monitored operation details in a log file.

[0824] Input: The captured operation event data.

[0825] Data processing: Writes operation event data to a log file.

[0826] Output: A log file containing the operations.

[0827] Specific operation: The user clicking the "New" button is recorded as "2023-10-01 10:00:00 - Microsoft Word - Clicked the New button."

[0828] Step 3:

[0829] Take a screenshot

[0830] The device will automatically take screenshots at appropriate times depending on the user's actions.

[0831] Input: The captured operation event data.

[0832] Data Processing: Capture screenshots immediately after the operation is completed and save them as image files.

[0833] Output: Image files of the captured screenshots.

[0834] Specific behavior: Capture the screen immediately after the user clicks the "Create New" button and save it as "screenshot_20231001_100000.png".

[0835] Step 4:

[0836] Generate descriptive text

[0837] The server generates explanatory text based on the recorded operations and screenshots.

[0838] Input: Log file recording operation details, image file of captured screenshots.

[0839] Data processing: Generative AI models (e.g., OpenAI's GPT-4) are used to create text that explains operating procedures in natural language.

[0840] Output: The generated description text.

[0841] Specific operation: In response to the action of clicking the "Create New" button, the server generates the text "Click the New button in Microsoft Word to create a new document."

[0842] Step 5:

[0843] Configuring the manual

[0844] The server combines the acquired screenshots with the generated explanatory text to create each page of the manual.

[0845] Input: Image files of captured screenshots, generated description text.

[0846] Data processing: Screenshots and corresponding explanatory text are laid out in HTML or PDF format and saved as a manual.

[0847] Output: A file with the configured manual.

[0848] Specific operation: The server combines "screenshot_20231001_100000.png" and the text "Click the New button in Microsoft Word to create a new document." into one page.

[0849] (Application example 1)

[0850] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0851] Conventional manual creation methods require manual recording of operating procedures, taking screenshots, and creating explanatory text, which is time-consuming and labor-intensive. Furthermore, recording the operating procedures of factory robots and generating detailed explanations is even more complex and requires specialized knowledge, making efficient manual creation difficult. This makes it difficult to accurately record and share operating procedures, resulting in reduced work efficiency and an increased risk of operational errors.

[0852] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0853] In this invention, the server includes means for recognizing computer operation procedures, means for automatically generating screenshots and text based on the operation procedures, means for creating a manual using the generated screenshots and text, means for recording robot operation procedures and generating screenshots and explanatory text, and means for generating detailed explanations of the operation procedures using a generative AI model, thereby enabling automatic recording of operation procedures and generation of detailed explanations.

[0854] "Means for recognizing computer operation procedures" is a function for detecting the operations performed by a user on a computer and recording those procedures.

[0855] "Means for automatically generating screenshots and text" is a function that automatically creates screen images and their descriptions based on recognized operating procedures.

[0856] The "means for creating a manual" is a function for creating a document explaining an operating procedure using the generated screenshots and text.

[0857] The "means for recording the robot's operation procedures" is a function for detecting a series of operations performed by a factory robot and recording those procedures.

[0858] The "means for generating explanatory text" is a function that automatically creates text that explains a procedure based on a recorded operating procedure.

[0859] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate text or images.

[0860] The "means for generating detailed explanations of operating procedures" is a function that uses a generative AI model to automatically create detailed explanations for recorded operating procedures.

[0861] The system for implementing this invention has the function of recognizing computer operating procedures, automatically generating screenshots and text, and creating manuals. It also includes the function of recording operating procedures of factory robots and generating detailed instructions using a generative AI model.

[0862] The server uses software to detect user actions on the computer. This software monitors user actions in real time and records the user actions. Based on the recorded actions, it captures screenshots and generates corresponding explanatory text.

[0863] The server then automatically creates a manual that explains the operating procedures using the generated screenshots and text. This manual is structured to be easily understandable by the user, and a table of contents and index are also automatically generated.

[0864] Furthermore, to record the operating procedures of the factory robot, the server monitors the robot's movements and records the procedures. Based on the recorded procedures, screenshots and explanatory text are generated. A generative AI model is used to automatically create detailed descriptions of the operating procedures. This generative AI model is implemented using OpenAI's API.

[0865] As a concrete example, consider the case of recording the operation procedure for a robot to pick up part A and attach it to part B. The server monitors the robot's operation and captures a screenshot of the moment when part A is picked up. Next, using the generative AI model, it inputs the prompt sentence "Please generate a detailed explanation of the procedure for picking up part A and attaching it to part B," and generates a detailed explanatory text.

[0866] Example prompt sentence:

[0867] Robot operation: Pick up part A and attach it to part B. Generate a detailed description of this operation.

[0868] In this way, it becomes possible to automatically record operating procedures and generate detailed explanations, thereby realizing efficient manual creation.

[0869] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0870] Step 1:

[0871] The server recognizes the actions that users take on their computers.

[0872] Input: User actions

[0873] Data processing: detecting and recording operating procedures

[0874] Output: Recorded operation steps

[0875] Specific operation: The server uses software that monitors the operation procedure in real time and records the operations performed by the user.

[0876] Step 2:

[0877] The server captures screenshots based on the recorded operating procedures.

[0878] Input: Recorded operating procedures

[0879] Data processing: Screenshot capture

[0880] Output: Captured screenshot

[0881] Specific operation: The server captures images of the screen corresponding to each step of the operating procedure and saves them as screenshots.

[0882] Step 3:

[0883] The server generates explanatory text based on the recorded operating procedures.

[0884] Input: Recorded operating procedures

[0885] Data processing: generating explanatory text

[0886] Output: Generated description text

[0887] Specific operation: The server automatically generates explanatory text corresponding to each step of the operating procedure and saves it as text.

[0888] Step 4:

[0889] The server creates a manual using the generated screenshots and text.

[0890] Input: Screenshot, Description text

[0891] Data processing: Manual creation

[0892] Output: Created manual

[0893] Specific operation: The server automatically creates a document that explains the operation procedure by combining screenshots and explanatory text.

[0894] Step 5:

[0895] The server records the operating procedures of the factory robot.

[0896] Input: Robot control

[0897] Data processing: detecting and recording operating procedures

[0898] Output: Recorded operation steps

[0899] Specific operation: The server monitors the robot's operation and records the operation procedures in real time.

[0900] Step 6:

[0901] The server generates screenshots and explanatory text based on the recorded robot operation procedures.

[0902] Input: Recorded operating procedures

[0903] Data processing: Capture screenshots and generate explanatory text

[0904] Output: Captured screenshots, generated explanatory text

[0905] Specific operation: The server captures screen images corresponding to each step of the robot's operating procedure and generates detailed explanatory text using a generative AI model.

[0906] Step 7:

[0907] The server uses a generative AI model to generate detailed instructions for the operation procedure.

[0908] Input: Prompt for operation procedure

[0909] Data processing: Text generation using generative AI models

[0910] Output: Detailed description text

[0911] Specific operation: The server uses a generative AI model (e.g., OpenAI API) to generate detailed instructions for the operation procedure. Example prompt: "Robot operation procedure: Pick up part A and attach it to part B. Please generate a detailed explanation for this operation."

[0912] Step 8:

[0913] The server uses the generated screenshots and detailed explanatory text to create a manual that explains how to operate the robot.

[0914] Input: screenshots, detailed description text

[0915] Data processing: Manual creation

[0916] Output: Created manual

[0917] Specific operation: The server automatically creates a document that explains the robot's operating procedures by combining screenshots and detailed explanatory text.

[0918] Example 2

[0919] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0920] The traditional manual creation process requires a lot of time and effort, especially when it comes to organizing screenshots and text and creating tables of contents and indexes. It's also difficult to maintain the quality and consistency of the generated manuals. This creates a demand for efficient, high-quality manual creation.

[0921] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for a user to input data, a means for generating screenshots and text using a generative AI model, a means for creating a manual by combining the generated screenshots and text, a means for automatically generating a table of contents and index for the manual, and a means for outputting the completed manual in PDF format and providing it to the user. This enables efficient and high-quality manual creation.

[0922] The "means for the user to input data" is an interface that allows the user to provide the system with the information required to create the manual.

[0923] "Means for generating screenshots and text using a generative AI model" refers to a function that uses a generative AI model to automatically generate screenshots and text from data provided by a user.

[0924] The "means for creating a manual by combining the generated screenshots and text" is a function for organizing the generated screenshots and text and putting them together in the format of a manual.

[0925] "Means for automatically generating a table of contents and index for a manual" is a function that extracts the titles of each chapter and section in a manual, creates a table of contents along with page numbers, and extracts important keywords to create an index.

[0926] "Means for outputting the completed manual in PDF format and providing it to the user" refers to a function for saving the final manual in PDF format and providing it to the user so that it can be downloaded.

[0927] "A function to adjust screenshots to an appropriate size using image processing software" refers to a function that uses software to process screenshots to an appropriate size and resolution.

[0928] "The ability to format text using text editing software" is the ability to use software to format the generated text into an appropriate format.

[0929] "A function that uses natural language processing technology to extract important keywords, list the pages on which those keywords appear, and create an index" is a function that uses natural language processing technology to extract important keywords from text, list the pages on which those keywords appear, and create an index.

[0930] MODE FOR CARRYING OUT THE INVENTION

[0931] This invention is a system that allows users to input data, generates screenshots and text using a generative AI model, and combines them to create a manual. It also has the ability to automatically generate a table of contents and index for the manual and output the completed manual in PDF format.

[0932] 1. Program Generation

[0933] A user accesses the system's interface using a web browser. On the interface, the user enters a prompt and uploads the necessary screenshots. For example, the user can enter the prompt "Please create a manual that explains the installation procedure for new software. Please use the screenshots and text below." and upload screenshots of the installation procedure.

[0934] 2. Program Processing

[0935] The server receives the prompt text entered by the user and invokes a generative AI model (e.g., OpenAI's GPT-4). The generative AI model generates appropriate text based on the prompt text and processes images based on screenshots uploaded by the user. For example, it generates instructions for each step of the installation procedure.

[0936] The server then combines the generated screenshots with the text, using image processing software to scale the screenshots to the appropriate size and text editing software to format the text, placing the screenshot on each page with the corresponding description below it.

[0937] The server also automatically generates a table of contents and index for the manual. Using natural language processing (NLP) technology, it extracts the titles of each chapter and section and creates a table of contents along with page numbers. It also extracts important keywords and creates an index by listing the pages where those keywords appear. For example, it extracts keywords such as "install," "configure," and "complete," and organizes the pages where those keywords appear into an index.

[0938] Finally, the server outputs the completed manual in PDF format. The PDF file is provided as a downloadable link to the user. The user clicks this link to download the manual, which can then be printed or shared as needed. For example, a user clicks the link to download the PDF file and share it with a colleague.

[0939] 3. Specific Examples

[0940] As a concrete example, consider a case where a user wants to create a manual for the installation procedure of new software. The user inputs screenshots and descriptions of the installation procedure into a generative AI model.

[0941] Example prompt sentence:

[0942] "Please create a manual that explains the installation process for new software. Use the following screenshots and text."

[0943] By inputting this prompt into a generative AI model, the model can automatically generate a detailed installation manual based on screenshots and text, including screenshots and corresponding instructions for each step, as well as an automatically generated table of contents and index.

[0944] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0945] Step 1:

[0946] The user enters data. Specifically, the user accesses the system interface using a web browser, enters a prompt, and uploads the necessary screenshots. The entered prompt and screenshots are sent to the server. For example, the user might enter a prompt such as "Please create a manual explaining the installation procedure for new software. Please use the screenshots and text below," and upload screenshots of the installation procedure.

[0947] Step 2:

[0948] The server generates screenshots and text using a generative AI model. The server receives the prompt text entered by the user and calls a generative AI model (e.g., GPT-4). The generative AI model generates appropriate text based on the prompt text and processes the image based on the screenshot uploaded by the user. The output is the generated text and the processed screenshot. For example, it generates instructions corresponding to each step of the installation procedure.

[0949] Step 3:

[0950] The server combines the generated screenshots with the text to create the manual. It uses image processing software to scale the screenshots to the appropriate size and text editing software to format the text. Specifically, it places a screenshot on each page and places corresponding explanatory text below it. The output is the initial version of the manual.

[0951] Step 4:

[0952] The server automatically generates a table of contents and index for the manual. Using natural language processing (NLP) technology, the server extracts the titles of each chapter and section and creates a table of contents along with page numbers. It also extracts important keywords and creates an index by listing the pages where these keywords appear. The output is a manual with an added table of contents and index. For example, keywords such as "installation," "configuration," and "completion" are extracted, and the pages where these keywords appear are organized into an index.

[0953] Step 5:

[0954] The server outputs the completed manual in PDF format and provides it to the user. The server saves the final manual in PDF format and provides it as a download link for the user. The user clicks this link to download the manual and can print or share it as needed. As output, a download link is provided to the user. For example, the user clicks the link to download the PDF file and share it with a colleague.

[0955] (Application example 2)

[0956] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0957] Conventionally, recording and creating manuals for operating and maintenance procedures for factory robots has relied on manual work, which has been time-consuming and labor-intensive. Creating the manual's table of contents and indexes has also been done manually, which has been inefficient and prone to errors. Furthermore, there has been a lack of a way to provide the generated manuals in various output formats for easy distribution and sharing.

[0958] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0959] In this invention, the server includes means for recognizing computer operating procedures, means for automatically generating screenshots and text based on the operating procedures, means for creating manuals using the generated screenshots and text, means for providing the generated manuals in various output formats to facilitate distribution and sharing of the manuals, and means for automatically recording factory robot operating procedures and maintenance procedures. This automates the recording of factory robot operating procedures and maintenance procedures and manual creation, significantly reducing time and labor. Furthermore, automatic generation of tables of contents and indexes enables efficient manual creation with fewer errors. Furthermore, providing the generated manuals in various output formats facilitates distribution and sharing, enabling information to be transmitted quickly and accurately.

[0960] "Means for recognizing computer operating procedures" refers to a device or software that monitors computer operations and automatically identifies those operating procedures.

[0961] A "screenshot" is a technique for capturing an image of what is displayed on a computer screen.

[0962] "Means for automatically generating text" refers to a device or software for automatically generating explanatory text or instruction text based on operating procedures.

[0963] The "means for creating a manual" refers to a device or software for combining the generated screenshots with text to create a document that summarizes operating procedures and explanations.

[0964] "Various output formats" refers to ways to store and present information in different file formats, such as PDF, HTML, DOCX, etc.

[0965] The "means for facilitating distribution and sharing of manuals" refers to a device or software for easily distributing and sharing the generated manuals via email, cloud storage, etc.

[0966] A "factory robot" is a mechanical device used to automate work in a factory.

[0967] "Means for automatically recording operating procedures and maintenance procedures" refers to a device or software that automatically records the operation and maintenance processes of factory robots and makes them available for later reference.

[0968] The "function to automatically generate a table of contents and index" is a function that automatically extracts each item and keyword in a manual and creates a table of contents and index.

[0969] A system for implementing this invention automatically records the operating procedures and maintenance procedures of factory robots and automatically generates manuals using screenshots and text. A specific embodiment of this system will be described below.

[0970] Hardware and software used

[0971] Hardware: Factory robots, cameras, computers

[0972] Software: Python, PIL (Python Imaging Library), FPDF, pytesseract

[0973] Data processing and calculation

[0974] A means of recognizing computer operating procedures

[0975] To recognize the operating procedures of the factory robot, the server uses a camera to capture images of the robot's operating screen, which are then sent to a computer where the operating procedures are automatically identified.

[0976] A way to automatically generate screenshots and text

[0977] The server saves the captured image as a screenshot and automatically generates instructions and instructions based on the operation procedure by extracting text from the image using pytesseract and generating appropriate instructions using a generative AI model.

[0978] How to create a manual

[0979] The server combines the generated screenshots and text to create a manual that summarizes operation procedures and explanations. To do this, it uses FPDF to generate the manual in PDF format.

[0980] A means to provide a variety of output formats for easy distribution and sharing

[0981] The server saves the generated manuals in various output formats such as PDF, HTML, and DOCX for distribution and sharing via email or cloud storage.

[0982] A means of automatically recording operating and maintenance procedures for factory robots

[0983] The server automatically records the operation and maintenance processes of factory robots and makes them available for future reference, ensuring that operation and maintenance procedures are accurately recorded and reflected in manuals.

[0984] Specific examples

[0985] For example, when initializing a factory robot, the server uses a camera to capture a screenshot of the operation screen. Next, it uses pytesseract to extract text from the image and uses a generative AI model to automatically generate an explanatory text such as, "In this procedure, you will perform the initial setup of the robot." This information is then combined to generate a PDF manual using FPDF, which is then saved in cloud storage.

[0986] Prompt Sentence Examples

[0987] Use the "Robot Operation Manual Auto-Generation App" to add the following steps to the manual.

[0988] 1. Initial setup of the robot

[0989] 2. Calibrating the robot

[0990] In this way, factory robot operation and maintenance procedures can be efficiently recorded and manuals can be automatically generated, significantly reducing time and labor and enabling accurate and efficient manual creation.

[0991] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0992] Step 1:

[0993] The server takes a picture of the factory robot's operation screen with a camera and captures a screenshot. The input is the image taken by the camera, and the output is a screenshot image file. This image file is used in subsequent processing.

[0994] Step 2:

[0995] The server analyzes the captured screenshot image using pytesseract and extracts the text from the image. The input is the screenshot image file, and the output is the extracted text data. This text data is used to generate operating procedures and explanations.

[0996] Step 3:

[0997] The server uses a generative AI model to automatically generate appropriate explanations and instructions based on the extracted text data. The input is the extracted text data, and the output is the generated explanations and instructions. These explanations and instructions are used as the content of the manual.

[0998] Step 4:

[0999] The server combines the generated screenshots and explanatory text to create a PDF manual using FPDF. The input is the screenshot image file and the generated explanatory text, and the output is a PDF manual file. This manual file is used for subsequent distribution and sharing.

[1000] Step 5:

[1001] The server saves the generated PDF manuals in cloud storage and distributes them by email as needed. The input is the PDF manual file, and the output is saving to cloud storage and distributing by email. This makes it easy to distribute and share the manuals.

[1002] Step 6:

[1003] The server automatically records the operating procedures and maintenance procedures of the factory robots and makes them available for future reference. The input is the operation and maintenance process of the factory robots, and the output is the recorded operating and maintenance procedure data. This data is used as the contents of the manuals.

[1004] Step 7:

[1005] The server automatically generates a table of contents and index for the generated manual. The input is a PDF manual file, and the output is the final manual file with the table of contents and index added. This makes it easy to refer to the manual.

[1006] Example 3

[1007] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1008] Conventional manual creation systems require users to manually input content and format it, which takes time and effort. Furthermore, it is difficult to distribute or share the generated manuals, making efficient information transmission difficult. Furthermore, it is difficult to flexibly generate manuals that meet user requirements.

[1009] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1010] In this invention, the server includes a means for receiving a user's request, a means for inputting a prompt sentence into a generative AI model, a means for converting the generated manual into a specified format, and a means for providing the manual to the user, thereby enabling automatic generation of a manual according to a user's request and output in various formats.

[1011] "Means for receiving user requests" refers to an interface for sending prompts and instructions entered by the user to the system.

[1012] "Means for inputting prompt sentences into the generative AI model" refers to the function of inputting prompt sentences received from the user into the generative AI model in an appropriate format and performing processing to generate the contents of the manual.

[1013] "Means for converting the generated manual into a specified format" refers to a function for converting the contents of the manual output from the generative AI model into a format specified by the user (PDF, HTML, Word document, etc.).

[1014] "Means for providing the manual to the user" refers to a method for providing the generated manual to the user, and includes, for example, generating a download link or sending an email.

[1015] A "generative AI model" refers to an artificial intelligence model that generates text based on a user's prompt, such as a model that uses natural language processing technology.

[1016] A "prompt" is a piece of text containing instructions or requests that a user inputs to a generative AI model.

[1017] The present invention relates to a system for automatically generating a manual based on a user's request and providing it in a specified format. A specific embodiment of this system will be described below.

[1018] System configuration

[1019] This system mainly consists of three elements: a server, a terminal, and a user.

[1020] 1. Server:

[1021] The server receives requests from users, generates manuals using generative AI models, converts them into the specified format, and provides them. The server includes the following software and hardware:

[1022] Generative AI models: For example, models using natural language processing techniques (e.g., GPT-4) are used.

[1023] PDF generation libraries: For example, generate manuals in PDF format using LaTeX or ReportLab.

[1024] Web server: A web server (e.g., Apache, Nginx) that receives requests from users and returns the processing results.

[1025] 2. Terminal:

[1026] A terminal is a device through which a user accesses the system and enters prompts. A terminal includes the following software and hardware:

[1027] Web Browser: A browser that allows users to access the system's web interface.

[1028] 3. User:

[1029] The user is responsible for inputting instructions to generate a manual using the system. The user accesses the system using a terminal and inputs a prompt.

[1030] System Operation

[1031] 1. Receive the user request:

[1032] The user accesses the system's web interface using a terminal and inputs a prompt statement, such as "I would like you to create a basic work procedure manual for new employees. The contents should include attendance procedures, basic work flows, and company rules. The output format should be PDF."

[1033] 2. Input the prompt sentence into the generative AI model:

[1034] The server analyzes the prompt received from the user and inputs it into a generative AI model (e.g., GPT-4), which then generates the content of the manual based on the prompt.

[1035] 3. Convert the generated manual to the specified format:

[1036] The server receives the manual content output from the generative AI model and converts it into the format specified by the user (PDF, HTML, Word document, etc.) For example, if the PDF format is specified, the server generates a PDF file using LaTeX or ReportLab.

[1037] 4. Provide the manual to users:

[1038] The server provides the generated manual to the user. Specifically, it generates a download link for the generated PDF file and notifies the user. The user then accesses the download link using their terminal and downloads the generated manual.

[1039] Specific examples

[1040] For example, if a user enters a prompt such as "I would like you to create a basic work procedure manual for new employees. The contents should include attendance procedures, basic work flow, and company rules. The output format should be PDF," the server will process as follows:

[1041] 1. Receiving the prompt:

[1042] User-entered prompt: "Please create a basic work procedure manual for new employees. The contents should include attendance procedures, basic work flow, and company rules. The output format should be PDF."

[1043] 2. Generate manuals using generative AI models:

[1044] The server inputs the prompt sentence into GPT-4 and generates the contents of the business procedure manual.

[1045] 3. Output the manual in the specified format:

[1046] The server embeds the generated text into a LaTeX template and generates a PDF file.

[1047] 4. Provide the manual to users:

[1048] The server sends the user a download link for the generated PDF file, which the user clicks to save in their local storage.

[1049] In this way, the system generates a manual in response to a user's request and provides it in the specified format. This makes it easy to distribute and share the manual. The flow of the specification process in the third embodiment will be described with reference to FIG. 15.

[1050] Step 1:

[1051] Receives user requests

[1052] The user uses a terminal to access the system's web interface and input a prompt. For example, the input prompt might be, "Please create a basic work procedure manual for new employees. Please include attendance procedures, basic work flow, and company rules. Please output in PDF format." The terminal then sends this prompt to the server.

[1053] Input: The prompt text entered by the user

[1054] Output: The prompt sent to the server

[1055] Step 2:

[1056] Input the prompt sentence into the generative AI model

[1057] The server analyzes the received prompt sentence and prepares it for input to the generative AI model. Specifically, the prompt sentence is formatted appropriately and input to the generative AI model (e.g., GPT-4). The generative AI model generates the contents of the manual based on the prompt sentence.

[1058] Input: The prompt sent to the server

[1059] Output: The prompts input to the generative AI model and the contents of the generated manual

[1060] Step 3:

[1061] Convert the generated manual to the specified format

[1062] The server receives the manual content output from the generative AI model and converts it into the format specified by the user (PDF, HTML, Word document, etc.). For example, if PDF format is specified, the server generates a PDF file using LaTeX and a PDF generation library (e.g., ReportLab). Specifically, it embeds the generated text into a LaTeX template and generates a PDF file.

[1063] Input: Generated manual content

[1064] Output: Manual converted to the specified format (e.g. PDF file)

[1065] Step 4:

[1066] Providing manuals to users

[1067] The server provides the generated manual to the user. Specifically, it generates a download link for the generated PDF file and notifies the user. The user then accesses the download link using their terminal and downloads the generated manual.

[1068] Input: Manual converted to the specified format (e.g. PDF file)

[1069] Output: The download link provided to the user, and the downloaded manual

[1070] In this way, the system generates manuals according to user requests and provides them in the specified format, making it easy to distribute and share the manuals.

[1071] (Application example 3)

[1072] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1073] Conventional manual creation systems recognize operating procedures and automatically generate screenshots, but the output formats of the generated manuals are limited, making it difficult to flexibly distribute and share them according to user needs. Furthermore, there is a need to efficiently generate operation and maintenance manuals for complex equipment such as factory robots and provide them in an appropriate format. This can lead to reduced understanding of operating procedures and reduced maintenance efficiency, leading to work delays and errors.

[1074] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1075] In this invention, the server includes means for recognizing computer operation procedures, means for automatically generating screenshots and text based on the operation procedures, means for creating a manual using the generated screenshots and text, means for generating the manual using a generative AI model, and means for providing the generated manual in various output formats such as PDF, HTML, Word document, etc. This enables users to efficiently generate detailed manuals for operating and maintaining factory robots and distribute and share them in the required format.

[1076] "Means for recognizing computer operating procedures" refers to a device or software that has the function of automatically detecting and recording computer operating procedures.

[1077] "Means for automatically generating screenshots and text" refers to a device or software that has the function of automatically creating a screen image (screenshot) and its explanatory text (text) based on recognized operating procedures.

[1078] The "means for creating a manual" refers to a device or software that has the function of creating a document (manual) that summarizes operating procedures and explanations using the generated screenshots and text.

[1079] A "means for generating a manual using a generative AI model" is a device or software that has the function of automatically generating a manual based on input information using an artificial intelligence model.

[1080] "Means for providing in various output formats such as PDF, HTML, Word document, etc." refers to a device or software that has the function of outputting the generated manual in different formats such as PDF, HTML, Word document, etc., and providing it in the format selected by the user.

[1081] The system for implementing this invention has the function of recognizing computer operating procedures, automatically generating screenshots and text, and using them to create manuals.It also has the function of generating manuals using a generative AI model and providing them in various output formats such as PDF, HTML, and Word documents.

[1082] System Program

[1083] The program for this system works as follows: First, when a user inputs a computer operation procedure, the server recognizes the operation procedure. Next, screenshots and text are automatically generated based on the recognized operation procedure. The server then creates a manual using the generated screenshots and text.

[1084] Furthermore, a generative AI model is used to generate a detailed manual based on the input information. This generative AI model uses OpenAI's GPT-3, etc. The generated manual is output in the format selected by the user (PDF, HTML, Word document, etc.).

[1085] Hardware and software used

[1086] The system uses the following hardware and software:

[1087] Hardware: Servers, user devices (smartphones, PCs, etc.)

[1088] Software: OpenAI API, FPDF, python-docx, BeautifulSoup

[1089] Data processing and calculation

[1090] The server analyzes the operating procedures entered by the user and generates screenshots and text. It then uses a generative AI model to generate a detailed manual. The generated text is then converted into formats such as PDF, HTML, and Word documents. Specifically, the FPDF library is used to generate PDFs, the python-docx library to generate Word documents, and the BeautifulSoup library to generate HTML.

[1091] Specific examples

[1092] For example, to generate a manual that describes the operating procedures for a factory robot, the user enters the following prompt:

[1093] Please explain the operating procedures for the factory robot, including how to power it on, how to set it up initially, basic operation, and maintenance procedures.

[1094] By inputting this prompt into a generative AI model, a manual containing detailed operating procedures is generated and output in the specified format, allowing users to efficiently obtain detailed information on the operation and maintenance of factory robots and distribute and share it in the format of their choice.

[1095] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1096] Step 1:

[1097] The user inputs the operation procedure from the terminal.

[1098] Input: The user inputs a prompt statement regarding the operating procedure of the factory robot.

[1099] Specific operation: The user enters into the terminal's input form, "Please explain the operating procedures for the factory robot. Specifically, please explain in detail how to turn it on, how to perform initial setup, basic operating procedures, and maintenance procedures."

[1100] Output: The prompt text entered is sent to the server.

[1101] Step 2:

[1102] The server recognizes the operation procedure.

[1103] Input: The prompt text sent by the user.

[1104] Specific operation: The server analyzes the received prompt text and recognizes the operation procedure.

[1105] Output: Data of recognized operation steps.

[1106] Step 3:

[1107] The server automatically generates screenshots and text.

[1108] Input: The data of the recognized operation procedure.

[1109] Specific operation: The server automatically generates relevant screenshots and their explanatory text based on the operating procedures.

[1110] Output: Generated screenshots and text.

[1111] Step 4:

[1112] The server generates the manual using the generative AI model.

[1113] Input: Generated screenshot and text.

[1114] How it works: The server uses a generative AI model (e.g., OpenAI's GPT-3) to generate a detailed manual based on screenshots and text.

[1115] Output: The generated manual text data.

[1116] Step 5:

[1117] The server provides the generated manual in formats such as PDF, HTML, Word document, etc.

[1118] Input: Generated manual text data.

[1119] Specific behavior: The server uses the FPDF library to generate PDFs, the python-docx library to generate Word documents, and the BeautifulSoup library to generate HTML.

[1120] Output: Generated manual in formats such as PDF, HTML, Word document, etc.

[1121] Step 6:

[1122] The user downloads the generated manual from the terminal.

[1123] Input: Generated manuals in formats such as PDF, HTML, Word documents, etc.

[1124] Specific operation: The user selects the desired format of the manual from the terminal and clicks the download button.

[1125] Output: The manual downloaded to the user's device.

[1126] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1127] "Example 1"

[1128] In one embodiment of the present invention, an emotion engine is incorporated. This emotion engine recognizes the user's emotions and adjusts the content of the manual based on those emotions. Specifically, if the user is having difficulty with an operation, the emotion engine recognizes that emotion and adjusts the manual to make it easier for the user to understand, such as by providing more detailed explanations for the relevant part of the manual.

[1129] "Example 2"

[1130] The emotion engine also has the ability to adjust the manual output format based on the user's emotions. For example, if the user is impatient, the emotion engine will recognize that emotion and output the manual in a concise format so that the user can quickly obtain the information they need. Conversely, if the user wants a deeper understanding, the emotion engine will recognize that emotion and output the manual in a detailed format so that the user can fully understand the information they need.

[1131] "Example 3"

[1132] Furthermore, the emotion engine adjusts the distribution and sharing method of the manual depending on the user's emotions. For example, if the user is in a hurry, the emotion engine will recognize that emotion and distribute the manual via an immediate means such as email or messaging app. Conversely, if the user wants to take their time to gather information, the emotion engine will recognize that emotion and distribute the manual via a means that allows the user to receive the information more slowly, such as by mailing a printed copy.

[1133] The processing flow of each embodiment will be described below.

[1134] "Example 1"

[1135] Step 1: The user initiates an action on the computer.

[1136] Step 2: The emotion engine recognizes the user's emotion.

[1137] Step 3: Adjust the content of the manual based on the emotions recognized by the emotion engine.

[1138] Step 4: The adjusted manual is presented to the user.

[1139] "Example 2"

[1140] Step 1: The user initiates an action on the computer.

[1141] Step 2: The emotion engine recognizes the user's emotion.

[1142] Step 3: Adjust the manual output format based on the emotions recognized by the emotion engine.

[1143] Step 4: The adjusted format of the manual is presented to the user.

[1144] "Example 3"

[1145] Step 1: The user initiates an action on the computer.

[1146] Step 2: The emotion engine recognizes the user's emotion.

[1147] Step 3: Adjust how you distribute and share the manual based on the emotions identified by the emotion engine.

[1148] Step 4: The manual is distributed to users in a coordinated manner.

[1149] Example 1

[1150] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1151] Conventional manual creation systems require manual recording of user operations, taking screenshots, and inputting text, which is extremely time-consuming. Furthermore, if a user experiences difficulty with an operation, it is difficult to adjust the manual to reflect that experience. This makes it difficult to quickly create a manual that is easy for users to understand.

[1152] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1153] In this invention, the server includes means for monitoring user operations, means for taking screenshots based on the operations, means for recording the operation contents as text, means for generating a draft of a manual using the generated screenshots and text, and means for recognizing the user's emotions and adjusting the contents of the manual based on the emotions. This makes it possible to automatically record the user's operations and quickly create an easy-to-understand manual that is adjusted according to the user's emotions.

[1154] The "means for monitoring user operations" refers to a device or software that has the function of monitoring the operations in real time when a user operates specific software.

[1155] A "means for taking a screenshot" is a device or software that has the function of taking a screenshot of the screen corresponding to a user operation each time that operation is recognized.

[1156] The "means for recording operation content as text" refers to a device or software that has the function of recording operations performed by a user in text format.

[1157] A "means for generating a manual draft" is a device or software that has the function of automatically creating a draft of a manual that explains operating procedures using the generated screenshots and text.

[1158] The "means for recognizing user's emotions" refers to a device or software that has the function of analyzing the emotions felt by the user in response to an operation and recognizing those emotions.

[1159] The "means for adjusting the contents of the manual" refers to a device or software that has the function of adjusting the explanation contents of the manual based on the recognized user's emotions to make them easier for the user to understand.

[1160] This invention is a manual creation assistant system that recognizes operations performed by a user on a computer and automatically generates screenshots and text corresponding to those operations. A specific embodiment of this system will be described below.

[1161] First, the device monitors user operations. Specifically, operation recognition software (e.g., OBS Studio or Snagit) installed on the device captures user operations in real time. When a user operates a specific software, the device recognizes the operation and takes a corresponding screenshot.

[1162] Next, the device records the user's operation as text. For example, if the user selects "Open" from the "File" menu to open an image file, the device generates text such as "Select Open from the File menu to open an image file."

[1163] The server then generates a draft manual based on the screenshots and text sent from the device. The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate instructions for the operation procedures. For example, for the operation "Launch image processing software and open an image file," the server generates an explanation such as "Launch the image processing software. Next, select 'File' -> 'Open' from the menu and open the image file you want to edit."

[1164] Furthermore, the server uses an emotion engine (e.g., Affectiva or IBM Watson Tone Analyzer) to analyze the user's emotions. If the user is having difficulty with an operation, the emotion engine recognizes that emotion and the server provides more detailed explanations. For example, if the user is having trouble using the brush tool, the server adds a detailed explanation such as, "Select the brush tool. You can adjust the brush size and opacity to fill in more detailed areas."

[1165] As a concrete example, consider the case where a user edits an image using image processing software. When the user starts the image processing software and opens an image file, the device recognizes this, takes a screenshot, and records the operation as text. The server uses this data to generate a draft manual like the one below:

[1166] 1. Start the image processing software (Screenshot 1).

[1167] 2. Select "File" → "Open" from the menu and open the image file you want to edit (Screenshot 2).

[1168] 3. Click the "Create New Layer" button in the "Layers" panel at the bottom right (Screenshot 3).

[1169] 4. Select the Brush Tool from the left toolbar (Screenshot 4).

[1170] 5. Click and drag over the part of the image you want to fill (Screenshot 5).

[1171] If the emotion engine recognizes the user's confusion, the server refines the explanation as follows:

[1172] 5. Click and drag over the area of ​​the image you want to fill. You can adjust the brush size and opacity to fill in more detail. You can change the brush settings in the options bar at the top (Screenshot 6).

[1173] An example of a prompt to input to a generative AI model is as follows:

[1174] You are creating a manual that will guide users through the steps of editing images using image processing software. Please generate a description for each step based on the following steps:

[1175] 1. Launch the image processing software

[1176] 2. Open the image file

[1177] 3. Add layers

[1178] 4. Select the Brush Tool

[1179] 5. Fill in part of an image

[1180] In this way, a manual is automatically generated based on the user's operations.

[1181] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1182] Step 1:

[1183] The user operates a specific piece of software.

[1184] Input: User actions

[1185] Output: Operation event

[1186] Specific operation: The user starts the image processing software and opens an image file.

[1187] Step 2:

[1188] The terminal monitors the user's operations.

[1189] Input: Operation event

[1190] Output: Operation log

[1191] Specific operation: Operation recognition software (e.g., OBS Studio or Snagit) installed on the device captures user operations in real time and generates an operation log.

[1192] Step 3:

[1193] The device will take a screenshot based on your actions.

[1194] Input: Operation log

[1195] Output: Screenshot

[1196] Specific behavior: The moment a user opens an image in image processing software, the device takes a screenshot of that screen.

[1197] Step 4:

[1198] The device records the operations as text.

[1199] Input: Operation log

[1200] Output: Operation text

[1201] Specific behavior: When a user selects "Open" from the "File" menu to open an image file, the device generates text such as "Select Open from the File menu to open an image file."

[1202] Step 5:

[1203] The server generates a draft of the manual using the generated screenshots and text.

[1204] Input: Screenshot, Operation text

[1205] Output: Manual Draft

[1206] Specific operation: The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate instructions for the operation. For example, for the operation "Launch image processing software and open an image file," the server generates an instruction such as "Launch the image processing software. Next, select 'File' -> 'Open' from the menu and open the image file you want to edit."

[1207] Step 6:

[1208] The server recognizes the user's emotions and adjusts the content of the manual accordingly.

[1209] Input: User operation log, operation text

[1210] Output: Adjusted manual draft

[1211] Specific operation: The server uses an emotion engine (e.g., Affectiva or IBM Watson Tone Analyzer) to analyze the user's emotions. If the user is having difficulty with an operation, the emotion engine recognizes that emotion and the server provides more detailed explanations. For example, if the user is having trouble using the brush tool, the server adds a detailed explanation such as, "Select the brush tool. You can adjust the brush size and opacity to fill in finer details."

[1212] (Application example 1)

[1213] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1214] Conventional manual creation systems require users to record operating procedures and manually generate screenshots and text, which is time-consuming and labor-intensive. Furthermore, they provide uniform explanations without considering the user's feelings, which means they are unable to provide sufficient support to users who have difficulty operating the system. Operating factory robots, in particular, requires complex procedures and specialized knowledge, which often confuses users. This increases the risk of operating errors and reduced efficiency.

[1215] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1216] In this invention, the server includes means for recognizing computer operation procedures, means for automatically generating screenshots and text based on the operation procedures, means for creating a manual using the generated screenshots and text, means for recognizing a user's emotion, and means for adjusting the content of the manual based on the recognized emotion. This automatically records the user's operation procedures and provides appropriate explanations according to the emotion, thereby reducing operation errors and enabling efficient manual creation.

[1217] "Means for recognizing computer operation procedures" refers to technology that automatically detects the operations a user performs on a computer and records those procedures.

[1218] "Means for automatically generating screenshots and text" is a technology that automatically creates screen capture images and their explanatory text based on recognized operating procedures.

[1219] The "means for creating a manual" is a technology that automatically creates a manual in a format that can be referenced by users using the generated screenshots and text.

[1220] "Means for recognizing user emotions" refers to technology that detects emotions from a user's facial expressions and behavior and determines their emotional state.

[1221] "Means for adjusting the content of the manual based on recognized emotions" refers to a technology that dynamically changes the explanation content and level of detail of the manual depending on the user's emotional state.

[1222] A system for carrying out the present invention has the following configuration: a server includes means for recognizing computer operating procedures, means for automatically generating screenshots and text, means for creating a manual using the generated screenshots and text, means for recognizing a user's emotion, and means for adjusting the content of the manual based on the recognized emotion.

[1223] Hardware and software used

[1224] Hardware: Computer, camera (for emotion recognition)

[1225] Software: Python, OpenCV (for image processing), PyAutoGUI (for taking screenshots), EmotionRecognizer (a custom library for emotion recognition), TextGenerator (a custom library for text generation)

[1226] Processing flow

[1227] The server first recognizes the user's computer operation procedures. Specifically, it detects and records the user's operation procedures in real time when the user operates specific software. Next, it automatically generates screenshots and text based on the recognized operation procedures. Screenshots are taken using PyAutoGUI, and text is generated using TextGenerator.

[1228] The server then captures the user's facial expressions using a camera and recognizes their emotions using EmotionRecognizer. Based on the recognized emotions, the server adjusts the content of the generated manual. For example, if the user is confused, it adds detailed explanations to the text.

[1229] Specific examples

[1230] For example, when recording the operating procedures of a factory robot, the server recognizes the robot startup procedure, part installation procedure, and operation check procedure, and generates screenshots and text corresponding to each step. If the user is confused, adding detailed explanations can reduce operating errors.

[1231] Prompt Sentence Examples

[1232] "Write a program that records the robot's startup steps and adds detailed instructions if the operator is confused."

[1233] In this way, by automatically recording the user's operating procedures and providing appropriate explanations according to their emotions, it is possible to reduce operating errors and create manuals efficiently.

[1234] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1235] Step 1:

[1236] The server recognizes the user's computer operation procedures. Specifically, when the user operates specific software, the operation procedures are detected and recorded in real time. The input is the user's operation, and the output is the data of the recognized operation procedures. This data is used in subsequent processing.

[1237] Step 2:

[1238] The server automatically generates screenshots based on the recognized operation steps. Specifically, it uses PyAutoGUI to capture images of the screen corresponding to the operation steps. The input is the data of the operation steps recognized in step 1, and the output is the captured screenshot image.

[1239] Step 3:

[1240] The server automatically generates text based on the recognized operation procedure. Specifically, it uses TextGenerator to generate an explanation of the operation procedure. The input is the data of the operation procedure recognized in step 1, and the output is the generated explanation text.

[1241] Step 4:

[1242] The server captures the user's facial expressions using a camera and recognizes the user's emotions using EmotionRecognizer. The input is the user's facial expression image, and the output is the recognized emotion data. This emotion data is used in subsequent processing.

[1243] Step 5:

[1244] The server adjusts the content of the manual based on the recognized emotions. Specifically, if the user is confused, it adds a detailed explanation to the generated explanatory text. The input is the explanatory text generated in step 3 and the emotion data recognized in step 4, and the output is the adjusted explanatory text.

[1245] Step 6:

[1246] The server creates a manual using the generated screenshots and adjusted explanatory text. The input is the screenshot images obtained in step 2 and the explanatory text adjusted in step 5, and the output is the completed manual. This manual is provided in a format that can be referenced by the user.

[1247] Example 2

[1248] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1249] In conventional manual creation systems, recording operation procedures, generating screenshots, and collecting text are often done manually, resulting in low work efficiency. Furthermore, the manual output format cannot be adjusted based on the user's emotions, making it difficult to provide information that meets the user's needs. Furthermore, the lack of an automatic table of contents or index generation function makes the manual difficult to read, making it difficult to quickly find the information needed. To solve these issues, a system is needed that can automatically recognize operation procedures, automatically generate screenshots and text, adjust the output format based on the user's emotions, and automatically generate a table of contents and index.

[1250] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1251] In this invention, the server includes means for recognizing computer operating procedures, means for automatically generating screenshots and text based on the operating procedures, means for creating a manual using the generated screenshots and text, and means for analyzing a user's emotions and adjusting the output format of the manual based on the analysis results. This enables automatic recognition of operating procedures and automatic generation of screenshots and text, thereby realizing appropriate manual output according to the user's emotions. In addition, the automatic generation function of a table of contents and index improves the readability of the manual and enables users to quickly find necessary information.

[1252] The "means for recognizing computer operation procedures" is a function for detecting operations performed by a user on a computer and recording those procedures.

[1253] A "screenshot" is a technique for capturing an image of what is displayed on a computer screen.

[1254] "Text" refers to text information entered by the user or explanatory text automatically generated by the system.

[1255] The "means for creating a manual" is a function that combines the generated screenshots and text to automatically generate a manual in a format that is easy for the user to understand.

[1256] "Means for analyzing user emotions" refers to technology that infers emotions from the user's facial expressions, behavior, input content, etc., and performs analysis based on that information.

[1257] The "means for adjusting the output format of the manual based on the analysis results" is a function for changing the content and format of the manual according to the results of the user's emotion analysis.

[1258] "Function to automatically generate a table of contents or index" is a technology that automatically creates a table of contents or index based on the contents of a manual.

[1259] This invention is a system that recognizes the operating procedures performed by a user on a computer, automatically generates screenshots and text based on those procedures, and creates a manual using the generated screenshots and text.It also has the function of analyzing the user's emotions and adjusting the output format of the manual based on the analysis results.

[1260] Hardware and software used

[1261] This system uses the following hardware and software:

[1262] Terminal: A device such as a computer or smartphone that a user operates.

[1263] Server: A server for organizing data, creating manuals, and performing sentiment analysis.

[1264] Emotion engine: Software for analyzing user emotions.

[1265] Manual creation software: Software for creating manuals, such as LaTeX or Microsoft Word.

[1266] Data processing and calculation

[1267] 1. User operation records

[1268] When a user operates the software, the device records the operation in real time, specifically saving the buttons the user clicks and the contents of the text fields they enter as a log.

[1269] 2. Generate screenshots and text

[1270] The device will automatically generate screenshots based on user actions, for example, capturing the screen at the moment the user clicks the "Install" button.

[1271] At the same time, the device collects the text entered by the user and the explanation automatically generated by the system, for example, "Next, click the Install button."

[1272] 3. Sending and organizing data

[1273] The device sends the generated screenshot and text to the server.

[1274] The server then organizes the data it receives, associating screenshots with corresponding text and arranging them in chronological order.

[1275] 4. Creating a manual

[1276] The server creates manuals based on the organized data, using software such as LaTeX and Microsoft Word.

[1277] The server will place the screenshot and text in the appropriate location and format it for easy viewing, for example by placing a corresponding description below the screenshot.

[1278] 5. Automatic table of contents and index generation

[1279] The server automatically generates a table of contents and index based on the contents of the manual, allowing users to easily find the information they need.

[1280] For example, add an "Installation Instructions" section to the table of contents and automatically insert its page number.

[1281] 6. Adjusting output format using emotion engine

[1282] When a user browses the manual, the device analyzes the user's emotions using an emotion engine.

[1283] For example, if the user is impatient, the terminal recognizes this emotion and instructs the server to output a manual in a concise format.

[1284] Conversely, if the user feels that he or she wants to understand more deeply, the terminal instructs the server to output the manual in a detailed format.

[1285] Specific examples

[1286] For example, consider a case where a user wants to learn how to install new software. As the user proceeds with the installation, the device records the operation as screenshots, and simultaneously collects comments entered by the user and explanations automatically generated by the system. This data is sent to a server, which then organizes it and creates a manual. Furthermore, if the user is impatient, the emotion engine outputs a concise manual, or if the user wants a deeper understanding, it outputs a detailed manual.

[1287] Prompt Sentence Examples

[1288] Examples of prompts to be input to a generative AI model include:

[1289] "Write a manual that explains the installation process for new software. Be concise if users are impatient, or detailed if they want a deeper understanding."

[1290] By using this prompt sentence, the generative AI model can generate an appropriate manual that corresponds to the user's emotions.

[1291] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1292] Step 1:

[1293] User operation records

[1294] Input: What you do on the computer (clicks, typing text, etc.)

[1295] How it works: When a user interacts with the software, the device records that interaction in real time, including saving the buttons the user clicks and the contents of text fields they enter as logs.

[1296] Output: Operation log data

[1297] Step 2:

[1298] Screenshot and text generation

[1299] Input: Operation log data

[1300] How it works: The device automatically generates screenshots based on user actions. For example, it captures the screen at the moment the user clicks the "Install" button. At the same time, the device collects text entered by the user and an explanation automatically generated by the system. For example, it generates an explanation such as "Next, click the Install button."

[1301] Output: Screenshot image and text data

[1302] Step 3:

[1303] Sending and organizing data

[1304] Input: Screenshot image and text data

[1305] How it works: The device sends the generated screenshots and text to the server, which then organizes the data it receives, associating screenshots with their corresponding text and arranging them in chronological order.

[1306] Output: Organized screenshots and text data

[1307] Step 4:

[1308] Manual creation

[1309] Input: Organized screenshots and text data

[1310] How it works: The server creates a manual based on the organized data. This can be done using software such as LaTeX or Microsoft Word. The server places screenshots and text in the appropriate locations and formats them in a readable format. For example, it places corresponding explanatory text below each screenshot.

[1311] Output: Manual data

[1312] Step 5:

[1313] Automatic table of contents and index generation

[1314] Input: Manual Data

[1315] What it does: The server automatically generates a table of contents and index based on the contents of the manual, allowing users to easily find the information they need. For example, it adds an "Installation Instructions" section to the table of contents and automatically inserts its page number.

[1316] Output: Manual data with table of contents and index added

[1317] Step 6:

[1318] Adjusting output format with emotion engine

[1319] Input: User emotion data, manual data

[1320] How it works: When a user is viewing a manual, the device analyzes the user's emotions using an emotion engine. For example, if the user is anxious, the device will recognize this emotion and instruct the server to output the manual in a concise format. Conversely, if the user wants to understand more deeply, the device will instruct the server to output the manual in a detailed format.

[1321] Output: Manual data in a format that corresponds to the user's emotions

[1322] (Application example 2)

[1323] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1324] Conventional manual creation systems recognize operating procedures and automatically generate screenshots and text, but they are unable to adjust the output format based on the user's emotions, making it difficult to provide the information users need quickly and appropriately. Furthermore, creating manuals for operating and configuring factory robots requires flexible responses based on the operator's emotions, but conventional systems are inadequate in this regard.

[1325] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1326] In this invention, the server includes means for recognizing computer operation procedures, means for automatically generating screenshots and text based on the operation procedures, means for creating a manual using the generated screenshots and text, means for analyzing a user's emotions, and means for adjusting the output format of the manual based on the analyzed emotions, thereby making it possible to provide the manual in a format appropriate to the user's emotions.

[1327] The "means for recognizing computer operation procedures" is a function for detecting a series of operations executed on a computer and recording the procedures.

[1328] "Means for automatically generating screenshots and text" is a function for automatically creating screen images (screenshots) and corresponding explanatory text (text) based on operating procedures.

[1329] The "means for creating a manual" is a function for combining the generated screenshots and text to create a manual in a format that can be referenced by the user.

[1330] "Means for analyzing user emotions" refers to a function that infers emotions from the user's input and behavior and analyzes those emotions as data.

[1331] "Means for adjusting the output format of the manual based on emotions" is a function that changes the content and format of the manual according to the analyzed emotions of the user, and provides information to the user in the most optimal form.

[1332] The "function to automatically generate a table of contents and index" is a function that automatically creates a table of contents and index based on the contents of the manual, allowing users to quickly find the information they need.

[1333] The "function to provide in various output formats" refers to the function to output the generated manual in multiple formats, such as PDF, HTML, and text file, and provide it in a form that is easy for users to use.

[1334] A system for carrying out the present invention has the following configuration: a server includes means for recognizing computer operating procedures, means for automatically generating screenshots and text, means for creating a manual using the generated screenshots and text, means for analyzing a user's emotions, and means for adjusting the output format of the manual based on the analyzed emotions.

[1335] Program processing explanation

[1336] The server first recognizes the computer's operating procedures. This is done by monitoring operation logs and user input to detect specific operating procedures. Next, it automatically generates screenshots and text based on the recognized operating procedures. The screenshots are generated using a camera or screen capture software (e.g., OpenCV) to capture the screen during operation, and the text is generated using OCR technology (e.g., pytesseract) to explain the operation.

[1337] The generated screenshots and text are then combined by the manual creation tool to create a user-readable manual, which is then output in PDF format using a PDF generation library such as FPDF.

[1338] Furthermore, the server uses a sentiment analysis model (e.g., the sentiment-analysis pipeline in the transformers library) to analyze the user's emotions. It infers emotions from the user's input and behavior, and adjusts the output format of the manual based on those emotions. For example, if the user is impatient, it generates a concise manual, but if the user wants more information, it generates a detailed manual.

[1339] Specific examples

[1340] For example, if an operator is impatient while operating a factory robot, the emotion engine will recognize that emotion and generate a concise manual. Conversely, if the operator wants more detailed information, the emotion engine will recognize that emotion and generate a detailed manual.

[1341] Prompt Sentence Examples

[1342] "Generate an operating manual for a factory robot. Use screenshots and operation logs to adjust output based on emotions."

[1343] In this way, it is possible to provide a manual in a format appropriate to the user's feelings.

[1344] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1345] Step 1:

[1346] The server recognizes the operating procedures of the computer.

[1347] Input: User operation logs and input data.

[1348] Processing: Analyze the operation log and detect specific operation steps.

[1349] Output: A list of recognized operation steps.

[1350] Step 2:

[1351] The server automatically generates screenshots and text based on the recognized operating procedures.

[1352] Input: A list of recognized operating steps.

[1353] Processing: Use screen capture software (e.g., OpenCV) to capture the screen during operation, and use OCR technology (e.g., pytesseract) to generate text describing the operation.

[1354] Output: Screenshot image and corresponding text.

[1355] Step 3:

[1356] The server creates a manual using the generated screenshots and text.

[1357] Input: Screenshot image and corresponding text.

[1358] Processing: Use a PDF generation library such as FPDF to create a PDF manual that combines screenshots and text.

[1359] Output: Manual in PDF format.

[1360] Step 4:

[1361] The server analyzes the user's emotions.

[1362] Input: User input data and behavior logs.

[1363] Processing: Analyze user sentiment using a sentiment analysis model (e.g., the sentiment-analysis pipeline in the transformers library).

[1364] Output: User sentiment data.

[1365] Step 5:

[1366] The server adjusts the manual output format based on the analyzed sentiment.

[1367] Input: User emotion data and a PDF manual.

[1368] Processing: Change the content and format of the manual based on the emotion data. For example, generate a concise manual if the user is impatient, or a detailed manual if the user wants more information.

[1369] Output: Manual in adjusted format.

[1370] Step 6:

[1371] The server provides the final manual to the user.

[1372] Input: Manual in adjusted format.

[1373] Action: Distribute the manual to users and share it as needed.

[1374] Output: User-referenceable manual.

[1375] Example 3

[1376] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1377] Conventional manual creation systems have the problem that it is difficult to provide information that meets the needs of users because they do not provide a flexible distribution method that responds to the user's emotions and situation.In addition, it is difficult to distribute and share the generated manuals, making it difficult to communicate information efficiently.

[1378] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means. In this invention, the server includes a means for recognizing computer operation procedures, a means for automatically generating images and text, a means for creating a document using the generated images and text, a means for analyzing a user's emotions, a means for determining a document distribution method based on the analysis result, and a means for distributing the document in the determined method. This enables a flexible distribution method according to the user's emotions and situation, thereby realizing efficient information transmission.

[1379] "Means for recognizing computer operating procedures" refers to a device or software for monitoring computer operations and recording those operating procedures.

[1380] "Means for automatically generating images and text" means a device or software for automatically generating associated images and text based on recognized operating procedures.

[1381] A "means for creating a document" is a device or software that uses the generated images and text to create a document in a format that is usable by a user.

[1382] "Means for analyzing user emotions" refers to a device or software for inferring and analyzing emotions from user input and behavior.

[1383] The "means for determining a document distribution method based on the analysis results" refers to a device or software for selecting the optimal document distribution method based on the results of the user emotion analysis.

[1384] A "means for distributing a document in a determined manner" is a device or software for providing a document to a user in accordance with a selected distribution method.

[1385] This invention is a system that recognizes user operation procedures, automatically generates images and text to create documents, and analyzes user emotions to determine the optimal distribution method. Specific embodiments of this system are described below.

[1386] First, the user enters data about the contents of the manual. This data may come in the form of text files, images, tables, etc. For example, the user may upload a text file about how to use a new piece of software.

[1387] The server then receives the data entered by the user and stores it in the server's storage, ready for analysis. For example, the server stores the uploaded text file in a specific directory.

[1388] The server then analyzes the received data using natural language processing (NLP) techniques to understand the content of the data. For example, the server might use a Python NLP library (e.g., spaCy) to analyze the content of the text file and extract important keywords and phrases.

[1389] The server then generates a document based on the analysis results, outputting the document in the format specified by the user (PDF, HTML, Word document, etc.) For example, the server may generate a PDF document using LaTeX.

[1390] Furthermore, the server analyzes the user's emotions using an emotion engine, which infers emotions from the user's input and behavior. For example, the server uses TextBlob to analyze emotions such as "I'm in a hurry" or "I want to take my time to get information" from the user's input text.

[1391] The server then decides how to distribute the document based on the results of the emotion engine. For example, if the user is in a hurry, the server may decide to distribute the document via email or a messaging app.

[1392] Finally, the server distributes the document in the manner determined: for example, the server may send the generated PDF document as an email attachment to the user, or, if the user prefers to dig deeper, print the document and mail it.

[1393] Specific examples

[1394] Let's say a user needs a manual on how to use new software, and they want the information quickly.

[1395] 1. The user uploads a text file to the server with instructions on how to use the new software.

[1396] 2. The server saves the uploaded text file in a specific directory.

[1397] 3. The server uses a Python NLP library (e.g., spaCy) to parse the contents of the text file and extract important keywords and phrases.

[1398] 4. The server generates a PDF manual using LaTeX.

[1399] 5. The server uses TextBlob to parse the "hurry" sentiment from the user's input text.

[1400] 6. The server realizes that the user is in a hurry and decides to distribute the manual by email.

[1401] 7. The server attaches the generated PDF manual to an email and sends it to the user.

[1402] Prompt Sentence Examples

[1403] "Generate a manual in PDF format on how to use new software and email it to users in a hurry."

[1404] In this way, the server can generate documents that meet the needs of the user and distribute them in an appropriate manner. The flow of the specific processing in the third embodiment will be described with reference to FIG.

[1405] Step 1:

[1406] The user inputs data. The user uploads data such as text files, images, and tables related to the contents of the manual from their terminal to the server. The input data is, for example, a text file on how to use new software. The output is the input data saved on the server.

[1407] Step 2:

[1408] The server receives the data. The server saves the data uploaded by the user in a specific directory. The input is the text file uploaded by the user, and the output is a file saved in the server's storage. Specifically, the server checks the directory where the file is saved and saves the file.

[1409] Step 3:

[1410] The server analyzes the data. The server uses a Python NLP library (e.g., spaCy) to analyze the contents of the saved text file. The input is the text file saved on the server, and the output is the analysis results: important keywords and phrases. Specifically, the server reads the text file and analyzes the text using the NLP library.

[1411] Step 4:

[1412] The server generates a document. The server generates a document based on the analysis results. It outputs the document in the format specified by the user (PDF, HTML, Word document, etc.). The input is keywords or phrases resulting from the analysis, and the output is the generated document. Specifically, the server uses LaTeX to generate a PDF document.

[1413] Step 5:

[1414] The server applies the emotion engine. The server uses the emotion engine to analyze the user's emotion. The input is the user's input text and behavioral data, and the output is the analyzed emotion information. Specifically, the server uses TextBlob to analyze the emotion from the user's input text.

[1415] Step 6:

[1416] The server determines the distribution method. The server determines the distribution method for the document based on the results of the emotion engine. The input is the analyzed emotion information, and the output is the determined distribution method. In concrete terms, the server analyzes the emotion information and selects the optimal distribution method.

[1417] Step 7:

[1418] The server distributes the document. The server provides the document to the user in the determined manner. The input is the generated document and the determined distribution method, and the output is the document distributed to the user. Specifically, the server attaches the generated PDF document to the user's email and sends it. If necessary, the server also prints and mails the document.

[1419] (Application example 3)

[1420] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1421] Conventional manual generation systems lack the flexibility to respond to user emotions and situations, making it difficult to provide information in an appropriate format to users who want to learn at their own pace, while being unable to provide information in an immediate manner to users who are in a hurry.In addition, the distribution methods for generated manuals are limited, making it difficult to provide them in a variety of formats to meet user needs.

[1422] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1423] In this invention, the server includes means for recognizing computer operation procedures, means for automatically generating screenshots and text, means for creating a manual using the generated screenshots and text, means for recognizing a user's emotion, means for adjusting the method for distributing the manual based on the recognized emotion, and means for providing the generated manual in various output formats. This makes it possible to provide a manual in an optimal format according to the user's emotion and situation, making it possible to provide information in an immediate manner to users who are in a hurry and information in an appropriate format to users who want to learn at their own pace.

[1424] "Means for recognizing computer operating procedures" refers to a device or software that monitors computer operations and analyzes and recognizes those operating procedures.

[1425] "Means for automatically generating screenshots and text" refers to a device or software for automatically generating a captured image of a screen (screenshot) and corresponding text based on recognized operating procedures.

[1426] The "means for creating a manual" refers to a device or software that uses the generated screenshots and text to create a manual in a format that is easy for users to understand.

[1427] A "means for recognizing a user's emotions" is a device or software that analyzes a user's facial expressions, voice, actions, etc. to recognize their emotions.

[1428] The "means for adjusting the method of distributing the manual based on the recognized emotion" is a device or software for optimizing the method of distributing the manual according to the emotion of the user.

[1429] "Means for providing the generated manual in various output formats" refers to a device or software for outputting the generated manual in different formats, such as PDF, HTML, or Word document, and providing it to the user.

[1430] An embodiment of the present invention will be described below. The following system is an emotion-responsive manual generation system installed on a factory robot.

[1431] The server first uses a means for recognizing computer operation procedures to monitor user operations and analyze and recognize those operation procedures. Next, it uses a means for automatically generating screenshots and text to automatically generate screen capture images (screenshots) and corresponding text based on the recognized operation procedures. The generated screenshots and text are compiled into a manual in a format that is easy for users to understand using a means for creating a manual.

[1432] Furthermore, a means for recognizing user emotions is used to analyze the user's facial expressions, voice, actions, etc. to recognize their emotions. Based on the recognized emotions, a means for adjusting the method of distributing the manual is used to select the optimal distribution method according to the user's emotions. For example, if the user is in a hurry, the manual can be distributed via a highly immediate method such as email or a messaging app, but if the user wants to take their time to understand the information, a method such as mailing a printed manual can be used.

[1433] The generated manuals can be output in various formats, such as PDF, HTML, and Word documents, and provided to users, making it possible to provide information in a variety of formats according to user needs.

[1434] Specific hardware and software include the Emotion Recognition API (e.g., Microsoft Azure Emotion API) for the emotion engine, the Manual Generator Module (e.g., a custom module written in Python) for manual generation, and the Twilio API for SMS and SMTP for Email for the communication module.

[1435] For example, if the emotion engine determines that a worker is in a hurry, a PDF manual will be sent to the worker's smartphone instantly. If the emotion engine determines that the worker wants to take their time to learn, a printed manual will be sent by mail.

[1436] Examples of prompts for generative AI models include:

[1437] "If users are in a hurry, distribute the manual in a timely manner. For example, email the manual in PDF format. If users want to study at their own pace, mail the printed manual."

[1438] The above is an embodiment of the present invention.

[1439] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1440] Step 1:

[1441] The server uses a means for recognizing computer operation procedures to monitor user operations and analyze and recognize those operation procedures. Inputs include the user's operation log and interface event data. The output is a list of recognized operation procedures. In this step, the operation log is analyzed and event data is collected.

[1442] Step 2:

[1443] The server uses a means for automatically generating screenshots and text to automatically generate screen capture images (screenshots) and corresponding text based on the recognized operating procedures. The input includes the list of operating procedures obtained in step 1. The output is pairs of screenshot images and text. In this step, screen captures are obtained and text is generated.

[1444] Step 3:

[1445] The server uses the generated screenshots and text to create a user-friendly manual. The input includes the screenshot image and text pairs obtained in step 2. The output is the completed manual. In this step, the screenshots and text are integrated and the manual is formatted.

[1446] Step 4:

[1447] The server uses a means to recognize the user's emotions by analyzing the user's facial expressions, voice, and behavior. The input includes the user's facial expression data and voice data. The output is a label for the recognized emotion. In this step, the emotion recognition algorithm is applied and the data is analyzed.

[1448] Step 5:

[1449] The server uses a means to adjust the manual distribution method based on the recognized emotions. The input includes the emotion labels obtained in step 4. The output is the determination of the optimal distribution method. In this step, the selection of the distribution method is based on the emotion labels.

[1450] Step 6:

[1451] The server uses a means to provide the generated manual in various output formats, outputting it in different formats such as PDF, HTML, Word document, etc., and provides it to the user. The input includes the completed manual obtained in step 3 and the distribution method determined in step 5. The output is the manual provided to the user. In this step, the format of the manual is converted and distributed.

[1452] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1453] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1454] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1455] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1456] [Third embodiment]

[1457] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1458] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1459] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1460] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1461] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1462] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1463] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1464] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1465] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1466] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1467] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1468] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1469] "Example 1"

[1470] The present invention is a manual creation assistant system. This system recognizes the operations a user performs on a computer and automatically generates screenshots and text corresponding to those operations. Specifically, when a user operates a specific piece of software, the system recognizes the operating procedure and generates corresponding screenshots and explanatory text. This information forms the basis for creating a manual.

[1471] "Example 2"

[1472] Furthermore, this embodiment of the present invention has a function to automatically create a manual using the generated screenshots and text. Specifically, the generated screenshots and text are combined, organized, and compiled into a manual format. It also has a function to automatically generate a table of contents and index for the manual. This makes it possible to significantly reduce the effort required to create a manual.

[1473] "Example 3"

[1474] Furthermore, the present invention has the function of providing the generated manual in various output formats. Specifically, the manual can be output in a format required by the user, such as PDF, HTML, or Word document. This makes it easy to distribute and share the manual.

[1475] The processing flow of each embodiment will be described below.

[1476] "Example 1"

[1477] Step 1: The user operates a specific piece of software.

[1478] Step 2: The system recognizes the user's operating procedure.

[1479] Step 3: The system will generate screenshots and explanatory text corresponding to the operating steps.

[1480] "Example 2"

[1481] Step 1: Get the screenshots and text generated by the system.

[1482] Step 2: The system combines this information and compiles it into a manual format.

[1483] Step 3: The system automatically generates a table of contents and index for the manual.

[1484] "Example 3"

[1485] Step 1: Obtain the system-generated manual.

[1486] Step 2: The system outputs the manual in the format required by the user. For example,

[1487] DF, HTML, Word documents, etc.

[1488] Example 1

[1489] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1490] Traditional manual creation required manually recording operation procedures, taking screenshots, and creating explanatory text, which was extremely time-consuming and labor-intensive. Furthermore, there was a high possibility of human error when recording operation procedures and creating explanatory text. Furthermore, creating a table of contents and index for the generated manual, providing it in various output formats, and distributing and sharing it were also time-consuming and labor-intensive.

[1491] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1492] In this invention, the server includes a means for monitoring user operations, a means for recording the operation details, a means for taking screenshots in response to the operations, a means for generating explanatory text using a generative AI model, and a means for creating a manual using the generated screenshots and explanatory text. This allows for real-time monitoring of user operations, automatic generation of screenshots and explanatory text based on the operation procedures, and efficient and accurate creation of manuals. It also enables automatic generation of a table of contents and index for the manual, provision in various output formats, and easy distribution and sharing.

[1493] A "means for monitoring user operations" is a device or software that has the function of monitoring the operations performed by a user on a computer in real time and capturing operation events.

[1494] "Means for recording operation details" refers to a device or software that has the function of recording the operation details of the monitored user in a log file or database and saving them in a format that can be referenced later.

[1495] A "means for taking a screenshot" is a device or software that has the function of capturing an image of the screen at an appropriate time in response to a user's operation and saving it as an image file.

[1496] A "means for generating explanatory text using a generative AI model" is a device or software that has the function of automatically generating explanatory text in natural language using a generative AI model based on recorded operation content.

[1497] A "means for creating a manual" is a device or software that has the function of combining the generated screenshots with explanatory text to form each page of the manual and to create a complete manual including a table of contents and index.

[1498] The "function to automatically generate a table of contents and index" is a function that automatically generates a table of contents and index based on the contents of the manual, allowing users to easily search for the information they need.

[1499] "Ability to provide in various output formats" refers to the ability to output the generated manual in different file formats such as PDF, HTML, DOCX, etc., and provide it flexibly according to the user's needs.

[1500] "Functions that make distribution and sharing easy" refers to functions that allow the generated manuals to be easily distributed and shared via email, cloud storage, an internal network, etc.

[1501] MODE FOR CARRYING OUT THE INVENTION

[1502] This invention is a manual creation assistant system that recognizes user operations on a computer and automatically generates screenshots and explanatory text corresponding to those operations. This system monitors user operations in real time and automatically generates screenshots and explanatory text based on the operating procedures, enabling efficient and accurate manual creation.

[1503] Hardware and software used

[1504] 1. Terminal

[1505] A computer device that monitors and records user operations.

[1506] Specifically, it includes software that captures events such as user mouse clicks, keyboard input, and window switching.

[1507] 2. Server

[1508] A computer device that generates explanatory text based on recorded operations and screenshots to create manuals.

[1509] Use a generative AI model (e.g., OpenAI's GPT-4) to create text that explains operating procedures in natural language.

[1510] Data processing and calculation

[1511] 1. Monitor user actions

[1512] The terminal monitors the user's operations in real time when using specific software.

[1513] Specifically, the terminal captures events such as user mouse clicks, keyboard input, and window switches.

[1514] 2. Record your actions

[1515] The terminal records the monitored operation details in a log file.

[1516] Specifically, it includes the type of operation, a timestamp, and information about the window or application that is the target of the operation.

[1517] 3. Take a screenshot

[1518] The device will automatically take screenshots at appropriate times depending on the user's actions.

[1519] Specifically, a screenshot is captured immediately after the operation is completed and saved as an image file.

[1520] 4. Generate explanatory text

[1521] The server generates explanatory text based on the recorded operations and screenshots.

[1522] Use a generative AI model (e.g., OpenAI's GPT-4) to create text that explains operating procedures in natural language.

[1523] 5. Configuring the manual

[1524] The server combines the acquired screenshots with the generated explanatory text to create each page of the manual.

[1525] Specifically, screenshots and corresponding explanatory text are laid out in HTML or PDF format and saved as a manual.

[1526] Specific examples

[1527] The user performs image editing operations using image processing software.

[1528] The device monitors and records when a user opens the "Filter" menu and selects the "Blur" option.

[1529] Your device will take a screenshot of the moment you select the "Blur" option.

[1530] Based on the recorded actions and screenshots, the server generates the following explanatory text: "Open the 'Filter' menu and select the 'Blur' option."

[1531] The server combines screenshots and explanatory text to form one page of the manual.

[1532] Prompt Sentence Examples

[1533] "Please explain the steps a user would take to edit an image in your image processing software, including applying a filter."

[1534] In this way, manuals can be created efficiently by monitoring user operations and automatically generating screenshots and explanatory text.

[1535] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1536] Step 1:

[1537] Monitor user actions

[1538] The terminal monitors the user's operations in real time when using specific software.

[1539] Input: User interaction events such as mouse clicks, keyboard input, and window switching.

[1540] Data processing: Capture operation events and extract information about the operation type, timestamp, and the window or application being operated on.

[1541] Output: The captured operation event data.

[1542] Specific behavior: Monitors when a user clicks the "New" button in Microsoft Word.

[1543] Step 2:

[1544] Record your operations

[1545] The terminal records the monitored operation details in a log file.

[1546] Input: The captured operation event data.

[1547] Data processing: Writes operation event data to a log file.

[1548] Output: A log file containing the operations.

[1549] Specific operation: The user clicking the "New" button is recorded as "2023-10-01 10:00:00 - Microsoft Word - Clicked the New button."

[1550] Step 3:

[1551] Take a screenshot

[1552] The device will automatically take screenshots at appropriate times depending on the user's actions.

[1553] Input: The captured operation event data.

[1554] Data Processing: Capture screenshots immediately after the operation is completed and save them as image files.

[1555] Output: Image files of the captured screenshots.

[1556] Specific behavior: Capture the screen immediately after the user clicks the "Create New" button and save it as "screenshot_20231001_100000.png".

[1557] Step 4:

[1558] Generate descriptive text

[1559] The server generates explanatory text based on the recorded operations and screenshots.

[1560] Input: Log file recording operation details, image file of captured screenshots.

[1561] Data processing: Generative AI models (e.g., OpenAI's GPT-4) are used to create text that explains operating procedures in natural language.

[1562] Output: The generated description text.

[1563] Specific operation: In response to the action of clicking the "Create New" button, the server generates the text "Click the New button in Microsoft Word to create a new document."

[1564] Step 5:

[1565] Configuring the manual

[1566] The server combines the acquired screenshots with the generated explanatory text to create each page of the manual.

[1567] Input: Image files of captured screenshots, generated description text.

[1568] Data processing: Screenshots and corresponding explanatory text are laid out in HTML or PDF format and saved as a manual.

[1569] Output: A file with the configured manual.

[1570] Specific operation: The server combines "screenshot_20231001_100000.png" and the text "Click the New button in Microsoft Word to create a new document." into one page.

[1571] (Application example 1)

[1572] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1573] Conventional manual creation methods require manual recording of operating procedures, taking screenshots, and creating explanatory text, which is time-consuming and labor-intensive. Furthermore, recording the operating procedures of factory robots and generating detailed explanations is even more complex and requires specialized knowledge, making efficient manual creation difficult. This makes it difficult to accurately record and share operating procedures, resulting in reduced work efficiency and an increased risk of operational errors.

[1574] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1575] In this invention, the server includes means for recognizing computer operation procedures, means for automatically generating screenshots and text based on the operation procedures, means for creating a manual using the generated screenshots and text, means for recording robot operation procedures and generating screenshots and explanatory text, and means for generating detailed explanations of the operation procedures using a generative AI model, thereby enabling automatic recording of operation procedures and generation of detailed explanations.

[1576] "Means for recognizing computer operation procedures" is a function for detecting the operations performed by a user on a computer and recording those procedures.

[1577] "Means for automatically generating screenshots and text" is a function that automatically creates screen images and their descriptions based on recognized operating procedures.

[1578] The "means for creating a manual" is a function for creating a document explaining an operating procedure using the generated screenshots and text.

[1579] The "means for recording the robot's operation procedures" is a function for detecting a series of operations performed by a factory robot and recording those procedures.

[1580] The "means for generating explanatory text" is a function that automatically creates text that explains a procedure based on a recorded operating procedure.

[1581] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate text or images.

[1582] The "means for generating detailed explanations of operating procedures" is a function that uses a generative AI model to automatically create detailed explanations for recorded operating procedures.

[1583] The system for implementing this invention has the function of recognizing computer operating procedures, automatically generating screenshots and text, and creating manuals. It also includes the function of recording operating procedures of factory robots and generating detailed instructions using a generative AI model.

[1584] The server uses software to detect user actions on the computer. This software monitors user actions in real time and records the user actions. Based on the recorded actions, it captures screenshots and generates corresponding explanatory text.

[1585] The server then automatically creates a manual that explains the operating procedures using the generated screenshots and text. This manual is structured to be easily understandable by the user, and a table of contents and index are also automatically generated.

[1586] Furthermore, to record the operating procedures of the factory robot, the server monitors the robot's movements and records the procedures. Based on the recorded procedures, screenshots and explanatory text are generated. A generative AI model is used to automatically create detailed descriptions of the operating procedures. This generative AI model is implemented using OpenAI's API.

[1587] As a concrete example, consider the case of recording the operation procedure for a robot to pick up part A and attach it to part B. The server monitors the robot's operation and captures a screenshot of the moment when part A is picked up. Next, using the generative AI model, it inputs the prompt sentence "Please generate a detailed explanation of the procedure for picking up part A and attaching it to part B," and generates a detailed explanatory text.

[1588] Example prompt sentence:

[1589] Robot operation: Pick up part A and attach it to part B. Generate a detailed description of this operation.

[1590] In this way, it becomes possible to automatically record operating procedures and generate detailed explanations, thereby realizing efficient manual creation.

[1591] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1592] Step 1:

[1593] The server recognizes the actions that users take on their computers.

[1594] Input: User actions

[1595] Data processing: detecting and recording operating procedures

[1596] Output: Recorded operation steps

[1597] Specific operation: The server uses software that monitors the operation procedure in real time and records the operations performed by the user.

[1598] Step 2:

[1599] The server captures screenshots based on the recorded operating procedures.

[1600] Input: Recorded operating procedures

[1601] Data processing: Screenshot capture

[1602] Output: Captured screenshot

[1603] Specific operation: The server captures images of the screen corresponding to each step of the operating procedure and saves them as screenshots.

[1604] Step 3:

[1605] The server generates explanatory text based on the recorded operating procedures.

[1606] Input: Recorded operating procedures

[1607] Data processing: generating explanatory text

[1608] Output: Generated description text

[1609] Specific operation: The server automatically generates explanatory text corresponding to each step of the operating procedure and saves it as text.

[1610] Step 4:

[1611] The server creates a manual using the generated screenshots and text.

[1612] Input: Screenshot, Description text

[1613] Data processing: Manual creation

[1614] Output: Created manual

[1615] Specific operation: The server automatically creates a document that explains the operation procedure by combining screenshots and explanatory text.

[1616] Step 5:

[1617] The server records the operating procedures of the factory robot.

[1618] Input: Robot control

[1619] Data processing: detecting and recording operating procedures

[1620] Output: Recorded operation steps

[1621] Specific operation: The server monitors the robot's operation and records the operation procedures in real time.

[1622] Step 6:

[1623] The server generates screenshots and explanatory text based on the recorded robot operation procedures.

[1624] Input: Recorded operating procedures

[1625] Data processing: Capture screenshots and generate explanatory text

[1626] Output: Captured screenshots, generated explanatory text

[1627] Specific operation: The server captures screen images corresponding to each step of the robot's operating procedure and generates detailed explanatory text using a generative AI model.

[1628] Step 7:

[1629] The server uses a generative AI model to generate detailed instructions for the operation procedure.

[1630] Input: Prompt for operation procedure

[1631] Data processing: Text generation using generative AI models

[1632] Output: Detailed description text

[1633] Specific operation: The server uses a generative AI model (e.g., OpenAI API) to generate detailed instructions for the operation procedure. Example prompt: "Robot operation procedure: Pick up part A and attach it to part B. Please generate a detailed explanation for this operation."

[1634] Step 8:

[1635] The server uses the generated screenshots and detailed explanatory text to create a manual that explains how to operate the robot.

[1636] Input: screenshots, detailed description text

[1637] Data processing: Manual creation

[1638] Output: Created manual

[1639] Specific operation: The server automatically creates a document that explains the robot's operating procedures by combining screenshots and detailed explanatory text.

[1640] Example 2

[1641] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1642] The traditional manual creation process requires a lot of time and effort, especially when it comes to organizing screenshots and text and creating tables of contents and indexes. It's also difficult to maintain the quality and consistency of the generated manuals. This creates a demand for efficient, high-quality manual creation.

[1643] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for a user to input data, a means for generating screenshots and text using a generative AI model, a means for creating a manual by combining the generated screenshots and text, a means for automatically generating a table of contents and index for the manual, and a means for outputting the completed manual in PDF format and providing it to the user. This enables efficient and high-quality manual creation.

[1644] The "means for the user to input data" is an interface that allows the user to provide the system with the information required to create the manual.

[1645] "Means for generating screenshots and text using a generative AI model" refers to a function that uses a generative AI model to automatically generate screenshots and text from data provided by a user.

[1646] The "means for creating a manual by combining the generated screenshots and text" is a function for organizing the generated screenshots and text and putting them together in the format of a manual.

[1647] "Means for automatically generating a table of contents and index for a manual" is a function that extracts the titles of each chapter and section in a manual, creates a table of contents along with page numbers, and extracts important keywords to create an index.

[1648] "Means for outputting the completed manual in PDF format and providing it to the user" refers to a function for saving the final manual in PDF format and providing it to the user so that it can be downloaded.

[1649] "A function to adjust screenshots to an appropriate size using image processing software" refers to a function that uses software to process screenshots to an appropriate size and resolution.

[1650] "The ability to format text using text editing software" is the ability to use software to format the generated text into an appropriate format.

[1651] "A function that uses natural language processing technology to extract important keywords, list the pages on which those keywords appear, and create an index" is a function that uses natural language processing technology to extract important keywords from text, list the pages on which those keywords appear, and create an index.

[1652] MODE FOR CARRYING OUT THE INVENTION

[1653] This invention is a system that allows users to input data, generates screenshots and text using a generative AI model, and combines them to create a manual. It also has the ability to automatically generate a table of contents and index for the manual and output the completed manual in PDF format.

[1654] 1. Program Generation

[1655] A user accesses the system's interface using a web browser. On the interface, the user enters a prompt and uploads the necessary screenshots. For example, the user can enter the prompt "Please create a manual that explains the installation procedure for new software. Please use the screenshots and text below." and upload screenshots of the installation procedure.

[1656] 2. Program Processing

[1657] The server receives the prompt text entered by the user and invokes a generative AI model (e.g., OpenAI's GPT-4). The generative AI model generates appropriate text based on the prompt text and processes images based on screenshots uploaded by the user. For example, it generates instructions for each step of the installation procedure.

[1658] The server then combines the generated screenshots with the text, using image processing software to scale the screenshots to the appropriate size and text editing software to format the text, placing the screenshot on each page with the corresponding description below it.

[1659] The server also automatically generates a table of contents and index for the manual. Using natural language processing (NLP) technology, it extracts the titles of each chapter and section and creates a table of contents along with page numbers. It also extracts important keywords and creates an index by listing the pages where those keywords appear. For example, it extracts keywords such as "install," "configure," and "complete," and organizes the pages where those keywords appear into an index.

[1660] Finally, the server outputs the completed manual in PDF format. The PDF file is provided as a downloadable link to the user. The user clicks this link to download the manual, which can then be printed or shared as needed. For example, a user clicks the link to download the PDF file and share it with a colleague.

[1661] 3. Specific Examples

[1662] As a concrete example, consider a case where a user wants to create a manual for the installation procedure of new software. The user inputs screenshots and descriptions of the installation procedure into a generative AI model.

[1663] Example prompt sentence:

[1664] "Please create a manual that explains the installation process for new software. Use the following screenshots and text."

[1665] By inputting this prompt into a generative AI model, the model can automatically generate a detailed installation manual based on screenshots and text, including screenshots and corresponding instructions for each step, as well as an automatically generated table of contents and index.

[1666] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1667] Step 1:

[1668] The user enters data. Specifically, the user accesses the system interface using a web browser, enters a prompt, and uploads the necessary screenshots. The entered prompt and screenshots are sent to the server. For example, the user might enter a prompt such as "Please create a manual explaining the installation procedure for new software. Please use the screenshots and text below," and upload screenshots of the installation procedure.

[1669] Step 2:

[1670] The server generates screenshots and text using a generative AI model. The server receives the prompt text entered by the user and calls a generative AI model (e.g., GPT-4). The generative AI model generates appropriate text based on the prompt text and processes the image based on the screenshot uploaded by the user. The output is the generated text and the processed screenshot. For example, it generates instructions corresponding to each step of the installation procedure.

[1671] Step 3:

[1672] The server combines the generated screenshots with the text to create the manual. It uses image processing software to scale the screenshots to the appropriate size and text editing software to format the text. Specifically, it places a screenshot on each page and places corresponding explanatory text below it. The output is the initial version of the manual.

[1673] Step 4:

[1674] The server automatically generates a table of contents and index for the manual. Using natural language processing (NLP) technology, the server extracts the titles of each chapter and section and creates a table of contents along with page numbers. It also extracts important keywords and creates an index by listing the pages where these keywords appear. The output is a manual with an added table of contents and index. For example, keywords such as "installation," "configuration," and "completion" are extracted, and the pages where these keywords appear are organized into an index.

[1675] Step 5:

[1676] The server outputs the completed manual in PDF format and provides it to the user. The server saves the final manual in PDF format and provides it as a download link for the user. The user clicks this link to download the manual and can print or share it as needed. As output, a download link is provided to the user. For example, the user clicks the link to download the PDF file and share it with a colleague.

[1677] (Application example 2)

[1678] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1679] Conventionally, recording and creating manuals for operating and maintenance procedures for factory robots has relied on manual work, which has been time-consuming and labor-intensive. Creating the manual's table of contents and indexes has also been done manually, which has been inefficient and prone to errors. Furthermore, there has been a lack of a way to provide the generated manuals in various output formats for easy distribution and sharing.

[1680] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1681] In this invention, the server includes means for recognizing computer operating procedures, means for automatically generating screenshots and text based on the operating procedures, means for creating manuals using the generated screenshots and text, means for providing the generated manuals in various output formats to facilitate distribution and sharing of the manuals, and means for automatically recording factory robot operating procedures and maintenance procedures. This automates the recording of factory robot operating procedures and maintenance procedures and manual creation, significantly reducing time and labor. Furthermore, automatic generation of tables of contents and indexes enables efficient manual creation with fewer errors. Furthermore, providing the generated manuals in various output formats facilitates distribution and sharing, enabling information to be transmitted quickly and accurately.

[1682] "Means for recognizing computer operating procedures" refers to a device or software that monitors computer operations and automatically identifies those operating procedures.

[1683] A "screenshot" is a technique for capturing an image of what is displayed on a computer screen.

[1684] "Means for automatically generating text" refers to a device or software for automatically generating explanatory text or instruction text based on operating procedures.

[1685] The "means for creating a manual" refers to a device or software for combining the generated screenshots with text to create a document that summarizes operating procedures and explanations.

[1686] "Various output formats" refers to ways to store and present information in different file formats, such as PDF, HTML, DOCX, etc.

[1687] The "means for facilitating distribution and sharing of manuals" refers to a device or software for easily distributing and sharing the generated manuals via email, cloud storage, etc.

[1688] A "factory robot" is a mechanical device used to automate work in a factory.

[1689] "Means for automatically recording operating procedures and maintenance procedures" refers to a device or software that automatically records the operation and maintenance processes of factory robots and makes them available for later reference.

[1690] The "function to automatically generate a table of contents and index" is a function that automatically extracts each item and keyword in a manual and creates a table of contents and index.

[1691] A system for implementing this invention automatically records the operating procedures and maintenance procedures of factory robots and automatically generates manuals using screenshots and text. A specific embodiment of this system will be described below.

[1692] Hardware and software used

[1693] Hardware: Factory robots, cameras, computers

[1694] Software: Python, PIL (Python Imaging Library), FPDF, pytesseract

[1695] Data processing and calculation

[1696] A means of recognizing computer operating procedures

[1697] To recognize the operating procedures of the factory robot, the server uses a camera to capture images of the robot's operating screen, which are then sent to a computer where the operating procedures are automatically identified.

[1698] A way to automatically generate screenshots and text

[1699] The server saves the captured image as a screenshot and automatically generates instructions and instructions based on the operation procedure by extracting text from the image using pytesseract and generating appropriate instructions using a generative AI model.

[1700] How to create a manual

[1701] The server combines the generated screenshots and text to create a manual that summarizes operation procedures and explanations. To do this, it uses FPDF to generate the manual in PDF format.

[1702] A means to provide a variety of output formats for easy distribution and sharing

[1703] The server saves the generated manuals in various output formats such as PDF, HTML, and DOCX for distribution and sharing via email or cloud storage.

[1704] A means of automatically recording operating and maintenance procedures for factory robots

[1705] The server automatically records the operation and maintenance processes of factory robots and makes them available for future reference, ensuring that operation and maintenance procedures are accurately recorded and reflected in manuals.

[1706] Specific examples

[1707] For example, when initializing a factory robot, the server uses a camera to capture a screenshot of the operation screen. Next, it uses pytesseract to extract text from the image and uses a generative AI model to automatically generate an explanatory text such as, "In this procedure, you will perform the initial setup of the robot." This information is then combined to generate a PDF manual using FPDF, which is then saved in cloud storage.

[1708] Prompt Sentence Examples

[1709] Use the "Robot Operation Manual Auto-Generation App" to add the following steps to the manual.

[1710] 1. Initial setup of the robot

[1711] 2. Calibrating the robot

[1712] In this way, factory robot operation and maintenance procedures can be efficiently recorded and manuals can be automatically generated, significantly reducing time and labor and enabling accurate and efficient manual creation.

[1713] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1714] Step 1:

[1715] The server takes a picture of the factory robot's operation screen with a camera and captures a screenshot. The input is the image taken by the camera, and the output is a screenshot image file. This image file is used in subsequent processing.

[1716] Step 2:

[1717] The server analyzes the captured screenshot image using pytesseract and extracts the text from the image. The input is the screenshot image file, and the output is the extracted text data. This text data is used to generate operating procedures and explanations.

[1718] Step 3:

[1719] The server uses a generative AI model to automatically generate appropriate explanations and instructions based on the extracted text data. The input is the extracted text data, and the output is the generated explanations and instructions. These explanations and instructions are used as the content of the manual.

[1720] Step 4:

[1721] The server combines the generated screenshots and explanatory text to create a PDF manual using FPDF. The input is the screenshot image file and the generated explanatory text, and the output is a PDF manual file. This manual file is used for subsequent distribution and sharing.

[1722] Step 5:

[1723] The server saves the generated PDF manuals in cloud storage and distributes them by email as needed. The input is the PDF manual file, and the output is saving to cloud storage and distributing by email. This makes it easy to distribute and share the manuals.

[1724] Step 6:

[1725] The server automatically records the operating procedures and maintenance procedures of the factory robots and makes them available for future reference. The input is the operation and maintenance process of the factory robots, and the output is the recorded operating and maintenance procedure data. This data is used as the contents of the manuals.

[1726] Step 7:

[1727] The server automatically generates a table of contents and index for the generated manual. The input is a PDF manual file, and the output is the final manual file with the table of contents and index added. This makes it easy to refer to the manual.

[1728] Example 3

[1729] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1730] Conventional manual creation systems require users to manually input content and format it, which takes time and effort. Furthermore, it is difficult to distribute or share the generated manuals, making efficient information transmission difficult. Furthermore, it is difficult to flexibly generate manuals that meet user requirements.

[1731] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1732] In this invention, the server includes a means for receiving a user's request, a means for inputting a prompt sentence into a generative AI model, a means for converting the generated manual into a specified format, and a means for providing the manual to the user, thereby enabling automatic generation of a manual according to a user's request and output in various formats.

[1733] "Means for receiving user requests" refers to an interface for sending prompts and instructions entered by the user to the system.

[1734] "Means for inputting prompt sentences into the generative AI model" refers to the function of inputting prompt sentences received from the user into the generative AI model in an appropriate format and performing processing to generate the contents of the manual.

[1735] "Means for converting the generated manual into a specified format" refers to a function for converting the contents of the manual output from the generative AI model into a format specified by the user (PDF, HTML, Word document, etc.).

[1736] The "means for providing the manual to the user" refers to a method for providing the generated manual to the user, and includes, for example, generating a download link or sending an email.

[1737] A "generative AI model" refers to an artificial intelligence model that generates text based on a user's prompt, such as a model that uses natural language processing technology.

[1738] A "prompt" is a piece of text containing instructions or requests that a user inputs to a generative AI model.

[1739] The present invention relates to a system for automatically generating a manual based on a user's request and providing it in a specified format. A specific embodiment of this system will be described below.

[1740] System configuration

[1741] This system mainly consists of three elements: a server, a terminal, and a user.

[1742] 1. Server:

[1743] The server receives requests from users, generates manuals using generative AI models, converts them into the specified format, and provides them. The server includes the following software and hardware:

[1744] Generative AI models: For example, models using natural language processing techniques (e.g., GPT-4) are used.

[1745] PDF generation libraries: For example, generate manuals in PDF format using LaTeX or ReportLab.

[1746] Web server: A web server (e.g., Apache, Nginx) that receives requests from users and returns the processing results.

[1747] 2. Terminal:

[1748] A terminal is a device through which a user accesses the system and enters prompts. A terminal includes the following software and hardware:

[1749] Web Browser: A browser that allows users to access the system's web interface.

[1750] 3. User:

[1751] The user is responsible for inputting instructions to generate a manual using the system. The user accesses the system using a terminal and inputs a prompt.

[1752] System Operation

[1753] 1. Receive the user request:

[1754] The user accesses the system's web interface using a terminal and inputs a prompt statement, such as "I would like you to create a basic work procedure manual for new employees. The contents should include attendance procedures, basic work flows, and company rules. The output format should be PDF."

[1755] 2. Input the prompt sentence into the generative AI model:

[1756] The server analyzes the prompt received from the user and inputs it into a generative AI model (e.g., GPT-4), which then generates the content of the manual based on the prompt.

[1757] 3. Convert the generated manual to the specified format:

[1758] The server receives the manual content output from the generative AI model and converts it into the format specified by the user (PDF, HTML, Word document, etc.) For example, if the PDF format is specified, the server generates a PDF file using LaTeX or ReportLab.

[1759] 4. Provide the manual to users:

[1760] The server provides the generated manual to the user. Specifically, it generates a download link for the generated PDF file and notifies the user. The user then accesses the download link using their terminal and downloads the generated manual.

[1761] Specific examples

[1762] For example, if a user enters a prompt such as "I would like you to create a basic work procedure manual for new employees. The contents should include attendance procedures, basic work flow, and company rules. The output format should be PDF," the server will process as follows:

[1763] 1. Receiving the prompt:

[1764] User-entered prompt: "Please create a basic work procedure manual for new employees. The contents should include attendance procedures, basic work flow, and company rules. The output format should be PDF."

[1765] 2. Generate manuals using generative AI models:

[1766] The server inputs the prompt sentence into GPT-4 and generates the contents of the business procedure manual.

[1767] 3. Output the manual in the specified format:

[1768] The server embeds the generated text into a LaTeX template and generates a PDF file.

[1769] 4. Provide the manual to users:

[1770] The server sends the user a download link for the generated PDF file, which the user clicks to save in their local storage.

[1771] In this way, the system generates a manual in response to a user's request and provides it in the specified format. This makes it easy to distribute and share the manual. The flow of the specification process in the third embodiment will be described with reference to FIG. 15.

[1772] Step 1:

[1773] Receives user requests

[1774] The user uses a terminal to access the system's web interface and input a prompt. For example, the input prompt might be, "Please create a basic work procedure manual for new employees. Please include attendance procedures, basic work flow, and company rules. Please output in PDF format." The terminal then sends this prompt to the server.

[1775] Input: The prompt text entered by the user

[1776] Output: The prompt sent to the server

[1777] Step 2:

[1778] Input the prompt sentence into the generative AI model

[1779] The server analyzes the received prompt sentence and prepares it for input to the generative AI model. Specifically, the prompt sentence is formatted appropriately and input to the generative AI model (e.g., GPT-4). The generative AI model generates the contents of the manual based on the prompt sentence.

[1780] Input: The prompt sent to the server

[1781] Output: The prompts input to the generative AI model and the contents of the generated manual

[1782] Step 3:

[1783] Convert the generated manual to the specified format

[1784] The server receives the manual content output from the generative AI model and converts it into the format specified by the user (PDF, HTML, Word document, etc.). For example, if PDF format is specified, the server generates a PDF file using LaTeX and a PDF generation library (e.g., ReportLab). Specifically, it embeds the generated text into a LaTeX template and generates a PDF file.

[1785] Input: Generated manual content

[1786] Output: Manual converted to the specified format (e.g. PDF file)

[1787] Step 4:

[1788] Providing manuals to users

[1789] The server provides the generated manual to the user. Specifically, it generates a download link for the generated PDF file and notifies the user. The user then accesses the download link using their terminal and downloads the generated manual.

[1790] Input: Manual converted to the specified format (e.g. PDF file)

[1791] Output: The download link provided to the user, and the downloaded manual

[1792] In this way, the system generates manuals according to user requests and provides them in the specified format, making it easy to distribute and share the manuals.

[1793] (Application example 3)

[1794] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1795] Conventional manual creation systems recognize operating procedures and automatically generate screenshots, but the output format of the generated manuals is limited, making it difficult to flexibly distribute and share them according to user needs. Furthermore, there is a need to efficiently generate operation and maintenance manuals for complex equipment such as factory robots and provide them in an appropriate format. This can lead to reduced understanding of operating procedures and reduced maintenance efficiency, leading to work delays and errors.

[1796] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1797] In this invention, the server includes means for recognizing computer operation procedures, means for automatically generating screenshots and text based on the operation procedures, means for creating a manual using the generated screenshots and text, means for generating the manual using a generative AI model, and means for providing the generated manual in various output formats such as PDF, HTML, Word document, etc. This enables users to efficiently generate detailed manuals for operating and maintaining factory robots and distribute and share them in the required format.

[1798] "Means for recognizing computer operating procedures" refers to a device or software that has the function of automatically detecting and recording computer operating procedures.

[1799] "Means for automatically generating screenshots and text" refers to a device or software that has the function of automatically creating a screen image (screenshot) and its explanatory text (text) based on recognized operating procedures.

[1800] The "means for creating a manual" refers to a device or software that has the function of creating a document (manual) that summarizes operating procedures and explanations using the generated screenshots and text.

[1801] A "means for generating a manual using a generative AI model" is a device or software that has the function of automatically generating a manual based on input information using an artificial intelligence model.

[1802] "Means for providing in various output formats such as PDF, HTML, Word document, etc." refers to a device or software that has the function of outputting the generated manual in different formats such as PDF, HTML, Word document, etc., and providing it in the format selected by the user.

[1803] The system for implementing this invention has the function of recognizing computer operating procedures, automatically generating screenshots and text, and using them to create manuals.It also has the function of generating manuals using a generative AI model and providing them in various output formats such as PDF, HTML, and Word documents.

[1804] System Program

[1805] The program for this system works as follows: First, when a user inputs a computer operation procedure, the server recognizes the operation procedure. Next, screenshots and text are automatically generated based on the recognized operation procedure. The server then creates a manual using the generated screenshots and text.

[1806] Furthermore, a generative AI model is used to generate a detailed manual based on the input information. This generative AI model uses OpenAI's GPT-3, etc. The generated manual is output in the format selected by the user (PDF, HTML, Word document, etc.).

[1807] Hardware and software used

[1808] The system uses the following hardware and software:

[1809] Hardware: Servers, user devices (smartphones, PCs, etc.)

[1810] Software: OpenAI API, FPDF, python-docx, BeautifulSoup

[1811] Data processing and calculation

[1812] The server analyzes the operating procedures entered by the user and generates screenshots and text. It then uses a generative AI model to generate a detailed manual. The generated text is then converted into formats such as PDF, HTML, and Word documents. Specifically, the FPDF library is used to generate PDFs, the python-docx library to generate Word documents, and the BeautifulSoup library to generate HTML.

[1813] Specific examples

[1814] For example, to generate a manual that describes the operating procedures for a factory robot, the user enters the following prompt:

[1815] Please explain the operating procedures for the factory robot, including how to power it on, how to set it up initially, basic operation, and maintenance procedures.

[1816] By inputting this prompt into a generative AI model, a manual containing detailed operating procedures is generated and output in the specified format, allowing users to efficiently obtain detailed information on the operation and maintenance of factory robots and distribute and share it in the format of their choice.

[1817] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1818] Step 1:

[1819] The user inputs the operation procedure from the terminal.

[1820] Input: The user inputs a prompt statement regarding the operating procedure of the factory robot.

[1821] Specific operation: The user enters into the terminal's input form, "Please explain the operating procedures for the factory robot. Specifically, please explain in detail how to turn it on, how to perform initial setup, basic operating procedures, and maintenance procedures."

[1822] Output: The prompt text entered is sent to the server.

[1823] Step 2:

[1824] The server recognizes the operation procedure.

[1825] Input: The prompt text sent by the user.

[1826] Specific operation: The server analyzes the received prompt text and recognizes the operation procedure.

[1827] Output: Data of recognized operation steps.

[1828] Step 3:

[1829] The server automatically generates screenshots and text.

[1830] Input: The data of the recognized operation procedure.

[1831] Specific operation: The server automatically generates relevant screenshots and their explanatory text based on the operating procedures.

[1832] Output: Generated screenshots and text.

[1833] Step 4:

[1834] The server generates the manual using the generative AI model.

[1835] Input: Generated screenshot and text.

[1836] How it works: The server uses a generative AI model (e.g., OpenAI's GPT-3) to generate a detailed manual based on screenshots and text.

[1837] Output: The generated manual text data.

[1838] Step 5:

[1839] The server provides the generated manual in formats such as PDF, HTML, Word document, etc.

[1840] Input: Generated manual text data.

[1841] Specific behavior: The server uses the FPDF library to generate PDFs, the python-docx library to generate Word documents, and the BeautifulSoup library to generate HTML.

[1842] Output: Generated manual in formats such as PDF, HTML, Word document, etc.

[1843] Step 6:

[1844] The user downloads the generated manual from the terminal.

[1845] Input: Generated manuals in formats such as PDF, HTML, Word documents, etc.

[1846] Specific operation: The user selects the desired format of the manual from the terminal and clicks the download button.

[1847] Output: The manual downloaded to the user's device.

[1848] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1849] "Example 1"

[1850] In one embodiment of the present invention, an emotion engine is incorporated. This emotion engine recognizes the user's emotions and adjusts the content of the manual based on those emotions. Specifically, if the user is having difficulty with an operation, the emotion engine recognizes that emotion and adjusts the manual to make it easier for the user to understand, such as by providing more detailed explanations for the relevant part of the manual.

[1851] "Example 2"

[1852] The emotion engine also has the ability to adjust the manual output format based on the user's emotions. For example, if the user is impatient, the emotion engine will recognize that emotion and output the manual in a concise format so that the user can quickly obtain the information they need. Conversely, if the user wants a deeper understanding, the emotion engine will recognize that emotion and output the manual in a detailed format so that the user can fully understand the information they need.

[1853] "Example 3"

[1854] Furthermore, the emotion engine adjusts the distribution and sharing method of the manual depending on the user's emotions. For example, if the user is in a hurry, the emotion engine will recognize that emotion and distribute the manual via an immediate means such as email or messaging app. Conversely, if the user wants to take their time to gather information, the emotion engine will recognize that emotion and distribute the manual via a means that allows the user to receive the information more slowly, such as by mailing a printed copy.

[1855] The processing flow of each embodiment will be described below.

[1856] "Example 1"

[1857] Step 1: The user initiates an action on the computer.

[1858] Step 2: The emotion engine recognizes the user's emotion.

[1859] Step 3: Adjust the content of the manual based on the emotions recognized by the emotion engine.

[1860] Step 4: The adjusted manual is presented to the user.

[1861] "Example 2"

[1862] Step 1: The user initiates an action on the computer.

[1863] Step 2: The emotion engine recognizes the user's emotion.

[1864] Step 3: Adjust the manual output format based on the emotions recognized by the emotion engine.

[1865] Step 4: The adjusted format of the manual is presented to the user.

[1866] "Example 3"

[1867] Step 1: The user initiates an action on the computer.

[1868] Step 2: The emotion engine recognizes the user's emotion.

[1869] Step 3: Adjust how you distribute and share the manual based on the emotions identified by the emotion engine.

[1870] Step 4: The manual is distributed to users in a coordinated manner.

[1871] Example 1

[1872] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1873] Conventional manual creation systems require manual recording of user operations, taking screenshots, and inputting text, which is extremely time-consuming. Furthermore, if a user experiences difficulty with an operation, it is difficult to adjust the manual to reflect that experience. This makes it difficult to quickly create a manual that is easy for users to understand.

[1874] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1875] In this invention, the server includes means for monitoring user operations, means for taking screenshots based on the operations, means for recording the operation contents as text, means for generating a draft of a manual using the generated screenshots and text, and means for recognizing the user's emotions and adjusting the contents of the manual based on the emotions. This makes it possible to automatically record the user's operations and quickly create an easy-to-understand manual that is adjusted according to the user's emotions.

[1876] The "means for monitoring user operations" refers to a device or software that has the function of monitoring the operations in real time when a user operates specific software.

[1877] A "means for taking a screenshot" is a device or software that has the function of taking a screenshot of the screen corresponding to a user operation each time that operation is recognized.

[1878] The "means for recording operation content as text" refers to a device or software that has the function of recording operations performed by a user in text format.

[1879] A "means for generating a manual draft" is a device or software that has the function of automatically creating a draft of a manual that explains operating procedures using the generated screenshots and text.

[1880] The "means for recognizing user's emotions" refers to a device or software that has the function of analyzing the emotions felt by the user in response to an operation and recognizing those emotions.

[1881] The "means for adjusting the contents of the manual" refers to a device or software that has the function of adjusting the explanation contents of the manual based on the recognized user's emotions to make them easier for the user to understand.

[1882] This invention is a manual creation assistant system that recognizes operations performed ...

Claims

1. A means for recognizing a computer operation procedure; means for automatically generating screenshots and text based on the operation procedure; means for creating a manual using the generated screenshots and the text; A means for recognizing the user's emotions; and A means for adjusting a distribution method of the generated manual based on the recognized emotion; A means for providing the generated manual in a plurality of output formats; A system including:

2. 2. The system of claim 1, wherein the means for creating the manual comprises a function for automatically generating a table of contents and an index for the manual.

3. 2. The system according to claim 1, wherein the means for creating the manual comprises a function for enabling the created manual to be provided in a plurality of output formats.

Images