system

Abstract

We provide the system. [Solution] A means of uploading draft documents, A means for analyzing the aforementioned draft data and generating flow data, Means of receiving instructions from users, Means for updating the flow data based on the aforementioned instructions, A means of generating documents that reflect updated flow data, A means for displaying the aforementioned material to the user, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventional business processes and the creation / updating work of manuals relied on manual labor, requiring a great deal of time and effort. Also, when creating or updating flows for multiple similar cases, reusability was poor and it was difficult to maintain consistency. Furthermore, since conventional systems could not flexibly reflect user instructions, they were an inhibiting factor for efficient work.

Means for Solving the Problems

[0005] The present invention solves the above problems by the following means.

[0006] The objective is to provide a system that includes means for uploading draft data, means for analyzing the draft data to generate flow data, means for receiving instructions from a user, means for updating the flow data based on the instructions, means for generating a document that reflects the updated flow data, and means for displaying the document to the user.

[0007] This allows for the efficient generation and updating of business flows and manuals based on preliminary documents, and enables flexible incorporation of user instructions. Furthermore, it supports various document formats such as PDF, Word, and Excel, and can receive instructions in natural language, thereby improving user experience.

[0008] Understood. Below are definitions of key terms included in the claims.

[0009] A "draft document" is a document that outlines the basic structure and overview of a business process flow or manual during its initial creation.

[0010] "Uploading" is the operation of sending a file from a user's device to a server.

[0011] "Analysis" is the process of reading the content of given materials and interpreting their meaning and structure.

[0012] "Flow data" refers to a data model of the business process flow generated from analyzed draft data.

[0013] An "instruction" is a specific operation or change request that a user makes to the system.

[0014] "Updating" refers to the operation of modifying or adding to existing flow data based on instructions from the user.

[0015] "Documents" or "generated documents" refer to business process diagrams and manuals newly created based on updated flow data.

[0016] "Display" refers to the operation of visually outputting information on the screen of a terminal.

[0017] "Natural language" refers to a language that humans use in daily life (e.g., Japanese, English), and is not program code or specialized symbols.

Brief Explanation of Drawings

[0018] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0019] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a processor with a reference numeral (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0022] In the following embodiments, a RAM (Random Access Memory) with a reference numeral is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0023] In the following embodiments, a storage with a reference numeral is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0024] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0026] [First Embodiment]

[0027] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0028] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0031] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0034] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0038] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0039] This invention relates to a system that automatically generates and updates business flows and manuals based on initial draft documents created by a user. This system performs a series of processes including analysis of the draft documents, generation of flow data, reception of user instructions, updating of flow data, and generation and display of deliverables.

[0040] System Configuration

[0041] This system consists of the following main components:

[0042] 1. How to upload draft documents

[0043] 2. Means for analyzing draft data and generating flow data

[0044] 3. Means of receiving instructions from the user

[0045] 4. Means for updating flow data based on instructions

[0046] 5. Means for generating documents that reflect updated flow data

[0047] 6. Means of displaying materials to users

[0048] The program's specific operation

[0049] Upload of initial draft documents

[0050] Users upload draft documents through the system interface. Various document formats are accepted, including PDF, Word, and Excel files. The terminal then sends the files selected by the user to the server.

[0051] Data analysis and flow data generation

[0052] The server analyzes the received draft data. This analysis uses natural language processing technology to understand the context and structure of the data. Based on the analysis results, flow data is generated that shows each step of the business process and its sequence.

[0053] User change instructions

[0054] Users give instructions to the system through an interactive interface. This includes adding new steps, modifying or deleting existing steps, and more. Instructions are entered in natural language, so no special technical knowledge is required.

[0055] Updating flow data

[0056] The server analyzes user instructions and updates the flow data accordingly. For example, if an instruction to add a new step is given, flow data reflecting its content and location will be generated.

[0057] Generation and display of deliverables

[0058] Based on the updated flow data, new flowcharts and manuals are generated. This process utilizes libraries for graphically drawing flowcharts. The generated documents are sent to the terminal and displayed to the user. The user can review the displayed documents and, if necessary, provide instructions for additional changes.

[0059] Specific example

[0060] For example, if a user wants to create a project management flow for the first time, they upload a document (e.g., a PDF) containing basic project steps as an initial draft. The server analyzes this document, extracts each step, and generates flow data.

[0061] Next, the user enters instructions into the interactive interface, such as "Add a task to Phase 2." The server analyzes these instructions and updates the flow data accordingly. The updated flow diagram is then regenerated and displayed to the user. The user reviews the displayed flow diagram and provides final approval or additional instructions as needed.

[0062] Thus, the present invention makes it easy to automatically generate and update business flows and manuals based on draft materials, significantly reducing the effort required from users.

[0063] The following describes the processing flow.

[0064] Understood. The specific processing steps of the program are described below.

[0065] Step 1:

[0066] Users select and upload their initial draft documents (e.g., PDF, Word, or Excel files) through the system interface.

[0067] Step 2:

[0068] The terminal sends the draft data selected by the user to the server. The sent files are temporarily stored on the server.

[0069] Step 3:

[0070] The server analyzes the received raw data. Natural language processing techniques are used for the analysis to understand the context and structure of the data (e.g., headings, paragraphs, lists).

[0071] Step 4:

[0072] The server generates flow data, which is a data model of the business process, based on the analysis results. This data includes the order and content of each step.

[0073] Step 5:

[0074] The terminal displays the generated flow data to the user as a graphical flowchart. The user reviews this and enters instructions for changes or additions into the interactive interface.

[0075] Step 6:

[0076] The user inputs instructions in natural language through a conversational interface. For example, they might input instructions such as, "Add a new task after step 3."

[0077] Step 7:

[0078] The terminal sends the user's instructions to the server. The server receives the instructions from the user and analyzes their content using natural language processing.

[0079] Step 8:

[0080] The server updates the flow data based on the instructions it has analyzed. For example, it might add new tasks or modify existing steps as instructed.

[0081] Step 9:

[0082] The server regenerates new flowcharts and manuals based on the updated flow data. A graphical drawing library is used for this purpose.

[0083] Step 10:

[0084] Updated flowcharts and manuals are sent from the server to the terminal. The terminal displays them, and the user reviews the content.

[0085] Step 11:

[0086] If the user requests additional changes, steps 6 through 10 are repeated. Once the user finally approves, the updated flowchart and manual are finalized.

[0087] This series of steps allows users to efficiently create and update workflows and manuals.

[0088] (Example 1)

[0089] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0090] To improve operational efficiency and reduce workload, it is necessary to automatically generate and update subsequent workflows and manuals based on the basic documents created by users initially. However, with current systems, users need to spend a lot of time and effort updating workflows and manuals, which does not contribute to operational efficiency. Furthermore, generating these often requires technical knowledge, and the need for specialized skills is a challenge.

[0091] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0092] In this invention, the server includes means for uploading draft data, means for analyzing the draft data and generating flow data, means for receiving instructions from the user, means for generating deliverables based on the updated flow data, and means for displaying the generated deliverables to the user. This enables the automatic generation and updating of business flows and manuals based on draft data.

[0093] A "draft document" is a basic document that serves as the basis for the user's initial workflow and manual.

[0094] "Flow data" refers to data generated by analyzing draft documents, which shows each step and its sequence in a business process.

[0095] "Analysis methods" refer to methods for analyzing preliminary data to understand its context and structure, and generating flow data.

[0096] "Generation method" refers to a means of generating deliverables such as new business flows and manuals based on updated flow data.

[0097] "Natural language processing technology" is a technology that uses computers to analyze and understand human natural language.

[0098] A "graphical display library" is a software library used to visually display business flows and data.

[0099] A "document file" refers to files that include business-related document formats such as PDF, Word, and Excel files.

[0100] A "conversational interface" is a user interface that allows users to input instructions using natural language.

[0101] This invention relates to a system that automatically generates and updates business flows and manuals based on initial draft documents created by the user. This system performs a series of processes including uploading the draft documents, analyzing the documents, generating flow data, receiving user instructions, updating the flow data, and generating and displaying deliverables.

[0102] System Configuration

[0103] This system consists of the following main components:

[0104] 1. How to upload draft documents

[0105] 2. Means for analyzing draft data and generating flow data

[0106] 3. Means of receiving instructions from the user

[0107] 4. Means for updating flow data based on the above instructions

[0108] 5. Means for generating deliverables based on updated flow data

[0109] 6. Means for displaying the generated deliverables to the user.

[0110] The program's specific operation

[0111] Upload of initial draft documents

[0112] Users upload draft documents through the system interface. Various document formats are accepted, including PDF, Word, and Excel files. The terminal then sends the files selected by the user to the server.

[0113] Data analysis and flow data generation

[0114] The server analyzes the received data. Here, Python-based natural language processing technologies (e.g., NLTK and spaCy) are used to understand the context and structure of the data. Based on the analysis results, flow data is generated that shows each step of the business process and its sequence. The flow data is saved in JSON format.

[0115] Receiving user change requests

[0116] The user gives instructions to the system using an interactive interface. This interface consists of text boxes and selection menus and accepts instructions entered in natural language. The terminal sends the user's instructions to the server.

[0117] Updating flow data

[0118] The server analyzes user instructions and updates flow data accordingly. It uses natural language processing technology to understand instructions and performs operations (addition, deletion, modification) on the relevant flow data.

[0119] Generation and display of deliverables

[0120] Based on the updated flow data, new flowcharts and manuals are generated. This uses libraries for graphically rendering flowcharts (e.g., D3.js or Graphviz). The generated materials are sent to the terminal and displayed to the user.

[0121] Specific example

[0122] For example, if a user wants to create a project management flow, they upload a PDF document outlining basic project steps as an initial draft. The server analyzes the document, extracts each step, and generates flow data.

[0123] Next, the user enters instructions into the interactive interface, such as "Add a task to Phase 2." The server analyzes these instructions and updates the flow data accordingly. The updated flow diagram is then regenerated and displayed to the user.

[0124] Prompt text examples

[0125] "Please analyze this PDF document and generate flow data."

[0126] "Please add a new task to Phase 2 of the flowchart."

[0127] "Please generate and display the updated flowchart."

[0128] In this way, the system of the present invention enables the automatic generation and updating of business flows and manuals based on draft data, significantly reducing the effort required from the user.

[0129] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0130] System program processing steps

[0131] Step 1:

[0132] Upload of initial draft documents

[0133] Input: The user selects draft documents (PDF, Word, Excel files, etc.) and uploads them to the system.

[0134] Specific actions: The user clicks the file selection button on the interface and selects a document file. Then, they press the upload button.

[0135] Data processing: The terminal retrieves the binary data of the selected file and sends it to the server via an HTTP POST request.

[0136] Output: The server receives the uploaded file.

[0137] Step 2:

[0138] Analysis of preliminary data and generation of flow data

[0139] Input: Received draft data (binary data)

[0140] Specific operation: The server uses a natural language processing library (e.g., NLTK or spaCy) to analyze the text of the provided data.

[0141] Data processing: The server understands the context and structure of the document and extracts semantic elements (steps, tasks, important points, etc.).

[0142] Output: Based on the extracted information, the server generates flow data showing each step of the business flow and its sequence, and stores it in JSON format.

[0143] Step 3:

[0144] Receiving user change requests

[0145] Input: User instructions (text entered in natural language)

[0146] Specific operation: The user enters instructions into an interactive interface and presses the submit button.

[0147] Data processing: The terminal sends the entered text to the server.

[0148] Output: The server receives instructions (text) from the user.

[0149] Step 4:

[0150] Updating flow data

[0151] Input: User instruction text and current flow data (JSON format)

[0152] Specific operation: The server analyzes user instructions using natural language processing technology.

[0153] Data processing: Based on the instructions, perform operations such as adding, deleting, and modifying flow data (each step and its sequence).

[0154] Output: The server generates updated flow data (in JSON format).

[0155] Step 5:

[0156] Generation and display of deliverables

[0157] Input: Updated flow data (JSON format)

[0158] Specific operation: The server uses a graphical display library (e.g., D3.js or Graphviz) to generate new flowcharts and manuals.

[0159] Data processing: Convert flow data into a format that is easy to visualize (image file, PDF, etc.).

[0160] Output: The generated artifacts are sent to the terminal and displayed to the user.

[0161] Specific example

[0162] For example, if a user wants to create a project management flow, they upload a PDF document containing basic project steps as an initial draft. The server analyzes the document, extracts each step, and saves it as flow data in JSON format.

[0163] Next, the user enters "Add a task to Phase 2" into the interactive interface. The terminal sends this instruction to the server, which parses the instruction and updates the flow data. The updated flow diagram is then regenerated, sent to the terminal, and displayed to the user.

[0164] In this way, it becomes possible to automatically generate and update business flows and manuals based on preliminary documents, significantly reducing the effort required from users.

[0165] (Application Example 1)

[0166] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0167] In traditional factory operations, creating and updating work flows and operation manuals was a manual process requiring considerable effort and time. Furthermore, there was no system that could quickly generate work procedures based on initial drafts and update them in real time according to user instructions. This led to frequent inconsistencies in work flows and operational errors, making efficient work execution difficult. Additionally, there was a need for a system that could accurately understand and implement user instructions given in natural language.

[0168] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0169] In this invention, the server includes means for uploading draft data, means for analyzing the draft data to generate flow data, means for receiving instructions from a user, means for updating the flow data based on the instructions, means for generating data that reflects the updated flow data, and means for displaying the data to the user. The application installed on the industrial machine includes means for analyzing draft data uploaded by field workers using natural language processing technology to automatically generate business flows and operating procedures, and means for updating these business flows and operating procedures based on user instructions and graphically displaying the operation of the industrial machine and the business flow. This enables the automation of complex factory operations and operating procedures, and allows for the rapid generation and updating of business flows and manuals.

[0170] A "draft document" is a document used as an initial design for a certain workflow or procedure.

[0171] "Uploading" refers to the method of sending electronic files owned by a user to a server and importing them into the system.

[0172] "Methods for analyzing and generating flow data" refers to methods that analyze uploaded draft documents and automatically generate data on business flows and operating procedures.

[0173] "Means of receiving user instructions" refers to the methods by which the system accepts instructions that users input into it.

[0174] "Means for updating the flow data based on instructions" refers to a method of changing existing flow data to new content in accordance with instructions from the user.

[0175] "Means of generating documents that reflect updated flow data" refers to methods of creating documents that describe new business flows and operating procedures based on updated flow data.

[0176] "Means of displaying materials to users" refers to methods for showing users generated business flows and operating procedures.

[0177] "Applications installed on industrial machinery" refers to software that is integrated into robots and equipment used in factories to execute work flows and operating procedures.

[0178] "Natural language processing technology" refers to the technology that enables computers to understand the sentences and words that humans use in everyday life, and to automate the analysis and response processes.

[0179] "Draft documents uploaded by on-site workers" refers to initial design documents created by factory workers and sent to the system.

[0180] "Methods for automatically generating business flows and operating procedures" refers to methods in which a system creates business flows and operating procedures based on draft documents without manual intervention.

[0181] "Methods for updating business workflows and operating procedures" refers to methods of adding new information to the current business workflow or operating procedures, or modifying existing information, based on user instructions.

[0182] "Means of graphically displaying the operation of industrial machinery and business workflows" refers to methods of visually representing and presenting to users how to operate industrial machinery and the flow of work using diagrams and graphs.

[0183] This invention relates to a system that provides an application to be installed on industrial machinery used in a factory, and that automatically generates and updates business flows and operation manuals. This system performs a series of processes including uploading draft data, analysis, generation of flow data, receiving user instructions, updating flow data, and generating and displaying documents.

[0184] System Configuration

[0185] This system consists of the following main components:

[0186] 1. Method for uploading draft documents: Users upload draft documents (PDF, Word, Excel files, etc.) to the system. This upload function is provided through the user terminal interface.

[0187] 2. Means for analyzing draft data and generating flow data: The server analyzes the received draft data using natural language processing technology and generates flow data that shows each step of the business flow and its order. The NLP model used is "dbmdz / bert-large-cased-finetuned-conll03-english", which is part of the "transformers" library.

[0188] 3. Means of receiving user instructions: Users give instructions to the system through an interactive interface. These instructions are entered in natural language, making it usable even without special technical knowledge.

[0189] 4. Means for updating the flow data based on the instructions: The server analyzes the instructions from the user and updates the flow data accordingly. For example, if an instruction such as "Add a new step" is given, flow data reflecting the content and location of that instruction is generated.

[0190] 5. Means of generating documents that reflect updated flow data: Based on the updated flow data, new flowcharts and operation manuals are generated. Libraries and software for graphically drawing flowcharts are often used.

[0191] 6. Means for displaying the document to the user: The generated document is sent to the terminal and displayed to the user. The user can review the displayed document and issue additional modification instructions as needed.

[0192] Specific example

[0193] For example, when a factory worker sets up assembly procedures for a new module, they upload a PDF file containing the basic instructions. The server analyzes the PDF file, extracts each step, and generates flow data. The worker then enters an instruction into the interactive interface, such as "Add a new module inspection." The server analyzes this instruction and updates the flow data. The updated flow chart and work manual are regenerated and displayed to the worker. The worker reviews it and provides final approval or additional instructions as needed.

[0194] Example of a prompt

[0195] "We have uploaded the initial assembly instructions."

[0196] "Add a new step: Module inspection"

[0197] "Please remove the inspection step from the assembly procedure."

[0198] Thus, the present invention makes it easy to automatically generate and update work flows and operating procedures within a factory, significantly reducing the workload for users.

[0199] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0200] Step 1:

[0201] The user uploads draft documents. These documents can be in formats such as PDF, Word, or Excel. The terminal provides an interface for selecting and submitting these draft documents, and sends the selected files to the server. The input is the file uploaded by the user, and the output is the draft document sent to the server.

[0202] Step 2:

[0203] The server analyzes the received draft data. Natural language processing techniques are used for the analysis to understand the context and structure of the data. Specifically, the "dbmdz / bert-large-cased-finetuned-conll03-english" model from the "transformers" library is used to extract each step of the business flow from the draft data. The input is the received draft data, and the output is the generated flow data.

[0204] Step 3:

[0205] The server initially generates a business flow based on the generated flow data. Here, each step and its sequence are defined and visualized as a graphical flowchart. This flowchart is displayed to the user in subsequent steps. The input is the flow data, and the output is the visualized business flow diagram.

[0206] Step 4:

[0207] The user inputs instructions into the system through an interactive interface. These instructions are entered in natural language and include adding new steps, modifying or deleting existing ones, and more. The input consists of the user's instructions (prompts), and the output is the content of the received instructions.

[0208] Step 5:

[0209] The server analyzes the user's instructions and updates the flow data accordingly. For example, if the instruction is "Add a new step: module inspection," the flow data will be updated to reflect its content and location. The input is the user's instructions and the current flow data, and the output is the updated flow data.

[0210] Step 6:

[0211] The server generates new business process diagrams and operation manuals based on the updated flow data. New diagrams and procedures containing specific details are generated, and this information is visualized. The input is the updated flow data, and the output is the new business process diagrams and operation manuals.

[0212] Step 7:

[0213] The terminal displays the generated business process diagrams and operation manuals to the user. The user can review the displayed materials and issue instructions for further changes as needed. The input is the new business process diagrams and operation manuals, and the output is the visualized materials displayed to the user.

[0214] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0215] This invention relates to a system that automatically generates and updates business flows and manuals based on initial drafts created by the user, and further incorporates an emotion engine that recognizes the user's emotions to provide responses adapted to the user.

[0216] System Configuration

[0217] This system consists of the following main components:

[0218] 1. How to upload draft documents

[0219] 2. Means for analyzing draft data and generating flow data

[0220] 3. Means of receiving instructions from the user

[0221] 4. Means for updating flow data based on instructions

[0222] 5. Means for generating documents that reflect updated flow data

[0223] 6. Means of displaying materials to users

[0224] 7. Emotion engine that recognizes user emotions

[0225] 8. Means for adjusting the system's response based on recognized emotions.

[0226] The program's specific operation

[0227] Upload of initial draft documents

[0228] The user selects and uploads their initial draft document (e.g., PDF, Word, or Excel file) through the system interface. The terminal then sends this file to the server. The sent file is temporarily stored on the server.

[0229] Data analysis and flow data generation

[0230] The server analyzes the received draft data. This analysis uses natural language processing techniques to understand the context and structure of the data (e.g., headings, paragraphs, lists). Based on the analysis results, flow data, which is a data model of the business process, is generated.

[0231] User change instructions

[0232] Users give instructions to the system through an interactive interface. This includes adding new steps, modifying or deleting existing steps, and more. Instructions are entered in natural language, so no special technical knowledge is required.

[0233] Use of an emotion engine

[0234] The emotion engine analyzes the user's voice and facial expressions to recognize their emotions. For example, it collects the user's voice tone and facial expressions through the camera and microphone when they input instructions, and analyzes their emotions. This helps determine whether the user is stressed or satisfied.

[0235] Flow data updates and response adjustments

[0236] The server integrates user instructions with the results of the emotion engine's analysis and updates the flow data accordingly. For example, if the user is stressed, the system displays more detailed guidance or additional support messages. Conversely, if the user is satisfied, it provides a concise response.

[0237] Generation and display of deliverables

[0238] Based on the updated flow data, new flow charts and manuals are generated. This process utilizes graphical drawing libraries and other tools. The generated materials and system responses are sent to the terminal and displayed to the user. The user can review the displayed materials and, if necessary, provide instructions for additional changes.

[0239] Specific example

[0240] For example, if a user wants to create a project management flow for the first time, they upload a document (e.g., a PDF) containing basic project steps as an initial draft. The server analyzes this document, extracts each step, and generates flow data.

[0241] Next, the user enters instructions into the dialogue interface, such as "Add a task to Phase 2." The server analyzes these instructions and updates the flow data accordingly. The emotion engine then analyzes the user's voice and facial expressions, and if it determines that the user is satisfied, it displays the updated flow diagram along with a relatively simple response. Conversely, if the user is stressed, the system provides more detailed explanations and guidance.

[0242] Thus, the present invention makes it possible to automatically generate and update business flows and manuals based on draft data, and furthermore, by recognizing user emotions and adjusting responses, it is possible to provide a more user-friendly system.

[0243] The following describes the processing flow.

[0244] Understood. The specific steps of the process are outlined below.

[0245] Step 1:

[0246] Users select and upload their initial draft documents (e.g., PDF, Word, or Excel files) through the system interface.

[0247] Step 2:

[0248] The terminal sends the draft data selected by the user to the server. The sent files are temporarily stored on the server.

[0249] Step 3:

[0250] The server analyzes the received data. Natural language processing techniques are used for the analysis to understand the context and structure of the data (e.g., headings, paragraphs, lists). Based on the analysis results, a data model (flow data) of the business process is generated.

[0251] Step 4:

[0252] The terminal displays the generated flow data to the user as a graphical flowchart. The user then reviews it.

[0253] Step 5:

[0254] Users give instructions to the system through an interactive interface, which includes adding new steps, modifying or deleting existing ones, and more.

[0255] Step 6:

[0256] The emotion engine collects and analyzes the user's voice and facial expressions through input devices (microphone and camera). The emotion engine recognizes the user's emotional state (e.g., feeling stressed, feeling satisfied).

[0257] Step 7:

[0258] The device sends user instructions and emotion recognition results from the emotion engine to the server.

[0259] Step 8:

[0260] The server analyzes user instructions using natural language processing techniques and updates flow data accordingly. It also adjusts the system's response based on the results of the emotion engine (e.g., providing detailed guidance if the user is stressed).

[0261] Step 9:

[0262] The server generates new flowcharts and manuals based on the updated flow data. A graphical drawing library is used for this purpose.

[0263] Step 10:

[0264] Along with the newly generated flowcharts and manuals, responses tailored to the user's emotions are sent to the terminal. For example, a concise response if the user is satisfied, and a detailed guide if they are stressed.

[0265] Step 11:

[0266] The terminal displays the generated new flowcharts, manuals, and system responses to the user. The user can review the content and, if necessary, provide instructions for additional changes.

[0267] Step 12:

[0268] If the user provides additional instructions, steps 5 through 11 are repeated. Finally, once the user approves, the updated flowchart and manual are finalized.

[0269] This series of steps allows users to efficiently create and update workflows and manuals, and furthermore, the system's responses are adjusted based on sentiment recognition, resulting in a better user experience.

[0270] (Example 2)

[0271] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0272] Traditional automated systems for generating business flows and manuals have limitations in improving the user experience because they provide uniform processing responses without considering user emotions or stress levels. Furthermore, a lack of convenience was a problem, as users often needed technical knowledge to issue instructions. Additionally, the difficulty for users to intuitively understand the update status and results of flow data was also an issue.

[0273] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0274] In this invention, the server includes means for uploading draft data, means for analyzing the draft data to generate flow data, and means for receiving instructions from the user. This makes it possible to recognize the user's emotions and stress levels and adjust the system response accordingly. Specifically, by including the analysis of the user's voice and facial expressions and adjusting the response based on the recognized emotions, the flow data update process becomes more user-friendly and the user can easily operate it with an intuitive interface.

[0275] A "draft document" is a document file format used by a system as the basis for analysis and data generation.

[0276] "Flow data" refers to a data model of a business process flow generated based on information analyzed from preliminary documents.

[0277] "User instructions" refer to input given by the user to the system, including the addition of new steps and the modification or deletion of existing steps.

[0278] "Means of recognizing emotions" refers to the function of a system that analyzes the user's voice and facial expressions to identify the user's emotional state.

[0279] "Means for adjusting responses" refers to a function that dynamically applies response content, such as confirmation messages and guidelines from the system, based on the recognized emotions of the user.

[0280] "Means of displaying materials to the user" refers to an interface for clearly displaying generated and updated flow data and related materials on the user's terminal.

[0281] This invention relates to a system that automatically generates and updates business flows and manuals based on initial drafts created by the user, and further incorporates an emotion engine that recognizes the user's emotions to provide responses adapted to the user.

[0282] System Configuration

[0283] This system is composed of the following main components:

[0284] 1. Means for uploading tapping materials

[0285] 2. Means for analyzing tapping materials and generating flow data

[0286] 3. Means for receiving instructions from the user

[0287] 4. Means for updating flow data based on instructions

[0288] 5. Means for analyzing the user's voice and expression to recognize emotions

[0289] 6. Means for adjusting the system's response based on the recognized emotions

[0290] 7. Means for generating materials reflecting the updated flow data

[0291] 8. Means for displaying materials to the user

[0292] Specific Explanation of Program Processing

[0293] Upload of Initial Tapping Materials

[0294] The user selects and uploads the initially created tapping materials (e.g., PDF, Word, Excel files) through the terminal interface. The terminal sends this file to the server. The server saves the uploaded file in the temporary storage area.

[0295] Analysis of Materials and Generation of Flow Data

[0296] The server analyzes the received draft data. This analysis uses natural language processing techniques (e.g., SpaCy, NLTK) to understand the context and structure of the data (headings, paragraphs, lists, etc.). Based on the analysis results, flow data, which is a data model of the business process, is generated.

[0297] User change instructions

[0298] The user provides instructions to the system through an interactive interface. This includes adding new steps, modifying or deleting existing ones. Instructions are entered in natural language, so no special technical knowledge is required. The terminal receives the user's input and sends it to the server.

[0299] Use of the emotion engine

[0300] The server receives user voice and facial expression data from the terminal and recognizes the user's emotions using an emotion engine. The emotion engine uses voice analysis (e.g., speech-to-text services) and facial expression analysis (e.g., OpenCV).

[0301] Flow data updates and response adjustments

[0302] The server integrates user instructions and sentiment analysis results, updating flow data accordingly. If the user is experiencing stress, the system displays detailed guidance and additional support messages. Conversely, if the user is satisfied, it provides a concise response.

[0303] Generation and display of deliverables

[0304] Based on the updated flow data, new flowcharts and manuals are generated. This process utilizes graphical drawing libraries (e.g., D3.js). The generated materials and system responses are sent to the terminal and displayed to the user.

[0305] Specific example

[0306] For example, when a user wants to create a project management flow for the first time, the following steps are executed.

[0307] The user uploads a PDF document with basic project steps as the initial pitch document. The server analyzes the document, extracts each step, and generates flow data. Next, the user enters an instruction such as "Please add a task to Phase 2" into the interactive interface. The server analyzes this instruction and appropriately updates the flow data. After that, if the emotion engine analyzes the user's voice and expression and determines that the user is satisfied, an updated flow diagram is displayed along with a relatively simple response. Conversely, if the user is feeling stressed, the system provides more detailed explanations and guides.

[0308] Example of a prompt sentence for the generation AI model

[0309] "As the initial steps of project management, please generate the following flow data. If you want to add tasks to each step of Phase 1 and Phase 2, please give appropriate instructions. Also, please adjust the support message according to the user's emotion."

[0310] The above is the form for implementing the present invention.

[0311] The flow of the specific process in Example 2 will be described using FIG. 13.

[0312] Step 1:

[0313] Input: The pitch document (PDF, Word, Excel file) created by the user for the first time

[0314] Specific operation: The user clicks the "Upload" button from the interface of the terminal. The terminal sends the selected file to the server as an HTTP POST request.

[0315] Output: The uploaded file is saved to the server.

[0316] Step 2:

[0317] Input: Uploaded draft document

[0318] Specific operation: The server reads the received file and converts it into string data. Natural language processing techniques (e.g., SpaCy or NLTK) are used to analyze the document's context and structure (headings, paragraphs, lists, etc.).

[0319] Output: Flow data (JSON format) generated based on the analysis results.

[0320] Step 3:

[0321] Input: Flow data analysis results

[0322] Specific operation: The server displays initial flow data in the user interface. The user inputs instructions such as adding, modifying, or deleting steps in natural language through an interactive interface.

[0323] Output: User instructions are sent to the server via the terminal.

[0324] Step 4:

[0325] Input: User instructions

[0326] Specific operation: The server receives user instructions and analyzes the content of the instructions using natural language understanding (NLU) technology (e.g., the BERT model). Based on the analyzed instructions, the server updates the flow data.

[0327] Output: Updated flow data

[0328] Step 5:

[0329] Input: User voice and facial expression data

[0330] Specific operation: The device captures the user's voice and facial expressions through the microphone and camera and sends them to the server. The server converts the voice data into text using a speech-to-text service and recognizes emotions using facial expression analysis technology (e.g., OpenCV).

[0331] Output: Emotion recognition result

[0332] Step 6:

[0333] Input: Emotion recognition result

[0334] Specific operation: The server adjusts its response based on the emotion recognition results and user instructions. For example, if the user is stressed, it generates detailed guides or additional support messages; if the user is satisfied, it provides a concise response.

[0335] Output: Adjusted response

[0336] Step 7:

[0337] Input: Updated flow data

[0338] Specific operation: The server uses a graphical drawing library (e.g., D3.js) to generate new flowcharts and manuals based on the updated flow data.

[0339] Output: Generated flowcharts and manuals

[0340] Step 8:

[0341] Input: Generated flowcharts and manuals, and adjusted responses.

[0342] Specific operation: The server sends these deliverables to the terminal. The terminal displays the received materials and responses on the screen.

[0343] Output: Flowcharts, manuals, and system response messages displayed in a user-readable format.

[0344] The above describes the processing steps of this system.

[0345] (Application Example 2)

[0346] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0347] The creation and updating of conventional work flows and manuals is extremely time-consuming and difficult to address in a way that takes into account the emotions and stress levels of workers. As a result, work efficiency may decrease, and worker satisfaction may also decline. This problem is particularly serious in workplaces such as factories, where real-time flow updates and immediate feedback are required. This invention aims to solve these problems.

[0348] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for uploading draft data, means for analyzing the draft data to generate business flow data, means for receiving instructions from the user, means for updating the business flow data based on the instructions, means for recognizing the user's emotions using emotion recognition means, means for adjusting the response based on the recognized emotions, means for generating a document that reflects the updated business flow data, and means for displaying the document to the user. This enables the automatic and efficient generation and updating of business flows and manuals, as well as flexible responses that respond to the emotions of the workers.

[0349] A "draft document" is a document containing the basic content that was created initially.

[0350] "Business process flow data" refers to a data model that represents the flow and procedures of a business process.

[0351] "User instructions" refer to commands given by the user to the system, such as additions, deletions, and modifications.

[0352] "Emotion recognition means" refers to a function that analyzes the user's voice and facial expressions to recognize their emotions.

[0353] "Means of adjusting responses" refers to a function that optimizes the system's response based on recognized emotions.

[0354] "Means for generating documents" refers to a function that automatically creates new documents based on analyzed and updated flow data.

[0355] "Means of displaying materials to users" refers to a function that displays generated materials in a format that users can review.

[0356] This invention relates to a system that automatically generates and updates business flow data based on a draft document created by the user initially. The system incorporates a function that recognizes the user's emotions and adjusts the response accordingly.

[0357] System Configuration

[0358] This system consists of the following main components:

[0359] 1. How to upload draft documents

[0360] 2. A means of generating business flow data by analyzing draft documents.

[0361] 3. Means of receiving instructions from the user

[0362] 4. Means for recognizing a user's emotions using emotion recognition means

[0363] 5. Means of adjusting responses based on perceived emotions

[0364] 6. Means for generating documents that reflect updated business flow data

[0365] 7. Means for displaying the aforementioned materials to the user

[0366] Specific hardware and software to be used

[0367] hardware

[0368] Factory robots: Responsible for generating and displaying work instructions and documents.

[0369] Camera: Used to capture the user's facial expressions and recognize emotions.

[0370] Microphone: Used to capture the user's voice and recognize emotions and instructions.

[0371] Tablets and monitors: Used to visually display updated workflow data.

[0372] software

[0373] Natural language processing libraries (e.g., SpaCy): Used to analyze uploaded draft documents and generate business flow data.

[0374] Emotion recognition engine (e.g., Microsoft® Azure® Face API, Google® Cloud Speech-to-Text): Used to analyze and recognize the user's emotions.

[0375] Flow data generation and update libraries (e.g., Graphviz): Used to generate data models of business processes and to draw visual flow diagrams based on updated data.

[0376] Flow of operations

[0377] The user uploads a draft document (e.g., PDF, Word, Excel file) they initially created. This document is sent to the server by the system and temporarily stored. The server uses a natural language processing library to analyze the document and generate business flow data. Next, the user provides instructions through an interactive interface, which are entered in natural language. These instructions are then analyzed, and the business flow data is updated.

[0378] Furthermore, emotion recognition measures analyze the user's voice and facial expressions to recognize their emotions. Based on these results, the response is adjusted. For example, if the user is feeling stressed, a detailed support message is provided. On the other hand, if the user is satisfied, a concise response is provided.

[0379] Based on updated workflow data, new documents are generated and displayed on tablets and monitors by factory robots.

[0380] Examples of specific cases and prompt statements

[0381] For example, consider a scenario where a factory worker uploads a work procedure manual they created for the first time. After uploading a PDF as a draft document to the system, the system analyzes it and generates business flow data. When the worker gives instructions in natural language, such as "Please add a new task to the next step," these instructions are analyzed and the business flow data is updated accordingly.

[0382] The emotion recognition system analyzes the worker's voice and facial expressions, and if the worker is experiencing stress, the system displays detailed guidance or additional support messages. Conversely, if the worker is satisfied, it provides a simple response.

[0383] The updated flowchart is displayed on a tablet or monitor, allowing workers to review it and provide additional instructions as needed.

[0384] Example of a prompt

[0385] "Please upload the work procedure manual."

[0386] "Add a new task to the next step."

[0387] "We are analyzing the emotions of the workers through cameras and microphones."

[0388] "We are adjusting our response based on the analysis results."

[0389] Thus, the present invention enables the automatic and efficient generation and updating of business flows and manuals, and also allows for flexible responses that respond to the emotions of the workers.

[0390] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0391] Step 1:

[0392] The user uploads draft documents (e.g., PDF, Word, Excel files) via their device. These draft documents are sent to the server and temporarily stored. The input is the draft document in file format, and the output is the saved draft document file.

[0393] Step 2:

[0394] The server analyzes the uploaded draft data using a natural language processing library (e.g., SpaCy). The analysis generates business flow data. The input is the saved draft data file, and the output is the generated business flow data.

[0395] Specific operations include text extraction and contextual analysis of the draft materials.

[0396] Step 3:

[0397] The user inputs instructions in natural language through an interactive interface. These instructions include adding new steps, modifying existing steps, and deleting existing ones. The input is the user's instructions, and the output is the analysis result.

[0398] Step 4:

[0399] The server analyzes user instructions and updates the business flow data. Natural language processing is also used for this analysis. The input is the analyzed user instructions, and the output is the updated business flow data.

[0400] Specific actions include extracting instructions and modifying corresponding business flow data.

[0401] Step 5:

[0402] The server analyzes the user's voice and facial expressions using emotion recognition tools to recognize their emotions. This analysis uses emotion recognition engines (e.g., Microsoft Azure Face API, Google Cloud Speech-to-Text). The input is the user's voice and facial expression data, and the output is the recognized emotion.

[0403] Step 6:

[0404] The server adjusts its response based on the perceived emotion. Specifically, it provides a detailed support message if the user is stressed, and a concise response if they are satisfied. The input is the perceived emotion, and the output is the adjusted response.

[0405] Step 7:

[0406] The server generates new documents based on updated business process flow data and displays them on tablets and monitors via factory robots. A flow data generation and updating library (e.g., Graphviz) is used for generation. The input is the updated business process flow data, and the output is the generated documents (flowcharts and manuals).

[0407] Step 8:

[0408] The user reviews the displayed materials and provides additional instructions as needed. The input is the generated materials, and the output is the user's new instructions.

[0409] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0410] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0411] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0412] [Second Embodiment]

[0413] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0414] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0415] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0416] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0417] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0418] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0419] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0420] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0421] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0422] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0423] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0424] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0425] This invention relates to a system that automatically generates and updates business flows and manuals based on initial draft documents created by a user. This system performs a series of processes including analysis of the draft documents, generation of flow data, reception of user instructions, updating of flow data, and generation and display of deliverables.

[0426] System Configuration

[0427] This system consists of the following main components:

[0428] 1. How to upload draft documents

[0429] 2. Means for analyzing draft data and generating flow data

[0430] 3. Means of receiving instructions from the user

[0431] 4. Means for updating flow data based on instructions

[0432] 5. Means for generating documents that reflect updated flow data

[0433] 6. Means of displaying materials to users

[0434] The program's specific operation

[0435] Upload of initial draft documents

[0436] Users upload draft documents through the system interface. Various document formats are accepted, including PDF, Word, and Excel files. The terminal then sends the files selected by the user to the server.

[0437] Data analysis and flow data generation

[0438] The server analyzes the received draft data. This analysis uses natural language processing technology to understand the context and structure of the data. Based on the analysis results, flow data is generated that shows each step of the business process and its sequence.

[0439] User change instructions

[0440] Users give instructions to the system through an interactive interface. This includes adding new steps, modifying or deleting existing steps, and more. Instructions are entered in natural language, so no special technical knowledge is required.

[0441] Updating flow data

[0442] The server analyzes user instructions and updates the flow data accordingly. For example, if an instruction to add a new step is given, flow data reflecting its content and location will be generated.

[0443] Generation and display of deliverables

[0444] Based on the updated flow data, new flowcharts and manuals are generated. This process utilizes libraries for graphically drawing flowcharts. The generated documents are sent to the terminal and displayed to the user. The user can review the displayed documents and, if necessary, provide instructions for additional changes.

[0445] Specific example

[0446] For example, if a user wants to create a project management flow for the first time, they upload a document (e.g., a PDF) containing basic project steps as an initial draft. The server analyzes this document, extracts each step, and generates flow data.

[0447] Next, the user enters instructions into the interactive interface, such as "Add a task to Phase 2." The server analyzes these instructions and updates the flow data accordingly. The updated flow diagram is then regenerated and displayed to the user. The user reviews the displayed flow diagram and provides final approval or additional instructions as needed.

[0448] Thus, the present invention makes it easy to automatically generate and update business flows and manuals based on draft materials, significantly reducing the effort required from users.

[0449] The following describes the processing flow.

[0450] Understood. The specific processing steps of the program are described below.

[0451] Step 1:

[0452] Users select and upload their initial draft documents (e.g., PDF, Word, or Excel files) through the system interface.

[0453] Step 2:

[0454] The terminal sends the draft data selected by the user to the server. The sent files are temporarily stored on the server.

[0455] Step 3:

[0456] The server analyzes the received data. Natural language processing techniques are used for the analysis to understand the context and structure of the data (e.g., headings, paragraphs, lists).

[0457] Step 4:

[0458] The server generates flow data, which is a data model of the business process, based on the analysis results. This data includes the order and content of each step.

[0459] Step 5:

[0460] The terminal displays the generated flow data to the user as a graphical flowchart. The user reviews this and enters instructions for changes or additions into the interactive interface.

[0461] Step 6:

[0462] The user inputs instructions in natural language through a conversational interface. For example, they might input instructions such as, "Add a new task after step 3."

[0463] Step 7:

[0464] The terminal sends the user's instructions to the server. The server receives the instructions from the user and analyzes their content using natural language processing.

[0465] Step 8:

[0466] The server updates the flow data based on the instructions it has analyzed. For example, it might add new tasks or modify existing steps as instructed.

[0467] Step 9:

[0468] The server regenerates new flowcharts and manuals based on the updated flow data. A graphical drawing library is used for this purpose.

[0469] Step 10:

[0470] Updated flowcharts and manuals are sent from the server to the terminal. The terminal displays them, and the user reviews the content.

[0471] Step 11:

[0472] If the user requests additional changes, steps 6 through 10 are repeated. Once the user finally approves, the updated flowcharts and manuals are finalized.

[0473] This series of steps allows users to efficiently create and update workflows and manuals.

[0474] (Example 1)

[0475] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0476] To improve operational efficiency and reduce workload, it is necessary to automatically generate and update subsequent workflows and manuals based on the basic documents created by users initially. However, with current systems, users need to spend a lot of time and effort updating workflows and manuals, which does not contribute to operational efficiency. Furthermore, generating these often requires technical knowledge, and the need for specialized skills is a challenge.

[0477] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0478] In this invention, the server includes means for uploading draft data, means for analyzing the draft data and generating flow data, means for receiving instructions from the user, means for generating deliverables based on the updated flow data, and means for displaying the generated deliverables to the user. This enables the automatic generation and updating of business flows and manuals based on draft data.

[0479] A "draft document" is a basic document that serves as the basis for the user's initial workflow and manual.

[0480] "Flow data" refers to data generated by analyzing draft documents, which shows each step and its sequence in a business process.

[0481] "Analysis methods" refer to methods for analyzing preliminary data to understand its context and structure, and generating flow data.

[0482] "Generation method" refers to a means of generating deliverables such as new business flows and manuals based on updated flow data.

[0483] "Natural language processing technology" is a technology that uses computers to analyze and understand human natural language.

[0484] A "graphical display library" is a software library used to visually display business flows and data.

[0485] A "document file" refers to files that include business-related document formats such as PDF, Word, and Excel files.

[0486] A "conversational interface" is a user interface that allows users to input instructions using natural language.

[0487] This invention relates to a system that automatically generates and updates business flows and manuals based on initial draft documents created by the user. This system performs a series of processes including uploading the draft documents, analyzing the documents, generating flow data, receiving user instructions, updating the flow data, and generating and displaying deliverables.

[0488] System Configuration

[0489] This system consists of the following main components:

[0490] 1. How to upload draft documents

[0491] 2. Means for analyzing draft data and generating flow data

[0492] 3. Means of receiving instructions from the user

[0493] 4. Means for updating flow data based on the above instructions

[0494] 5. Means for generating deliverables based on updated flow data

[0495] 6. Means for displaying the generated deliverables to the user.

[0496] The specific operation of the program

[0497] Upload of initial draft documents

[0498] Users upload draft documents through the system interface. Various document formats are accepted, including PDF, Word, and Excel files. The terminal then sends the files selected by the user to the server.

[0499] Data analysis and flow data generation

[0500] The server analyzes the received data. Here, Python-based natural language processing technologies (e.g., NLTK or spaCy) are used to understand the context and structure of the data. Based on the analysis results, flow data is generated that shows each step of the business process and its sequence. The flow data is saved in JSON format.

[0501] Receiving user change requests

[0502] The user gives instructions to the system using an interactive interface. This interface consists of text boxes and selection menus and accepts instructions entered in natural language. The terminal sends the user's instructions to the server.

[0503] Updating flow data

[0504] The server analyzes user instructions and updates flow data accordingly. It uses natural language processing technology to understand instructions and performs operations (addition, deletion, modification) on the relevant flow data.

[0505] Output generation and display

[0506] Based on the updated flow data, new flowcharts and manuals are generated. This uses libraries for graphically rendering flowcharts (e.g., D3.js or Graphviz). The generated materials are sent to the terminal and displayed to the user.

[0507] Specific example

[0508] For example, if a user wants to create a project management flow, they upload a PDF document outlining basic project steps as an initial draft. The server analyzes the document, extracts each step, and generates flow data.

[0509] Next, the user enters instructions into the interactive interface, such as "Add a task to Phase 2." The server analyzes these instructions and updates the flow data accordingly. The updated flow diagram is then regenerated and displayed to the user.

[0510] Prompt text examples

[0511] "Please analyze this PDF document and generate flow data."

[0512] "Please add a new task to Phase 2 of the flowchart."

[0513] "Please generate and display the updated flowchart."

[0514] In this way, the system of the present invention enables the automatic generation and updating of business flows and manuals based on draft data, significantly reducing the effort required from the user.

[0515] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0516] System program processing steps

[0517] Step 1:

[0518] Upload of initial draft documents

[0519] Input: The user selects draft documents (PDF, Word, Excel files, etc.) and uploads them to the system.

[0520] Specific action: The user clicks the file selection button on the interface and selects a document file. Then, they press the upload button.

[0521] Data processing: The terminal retrieves the binary data of the selected file and sends it to the server via an HTTP POST request.

[0522] Output: The server receives the uploaded file.

[0523] Step 2:

[0524] Analysis of preliminary data and generation of flow data

[0525] Input: Received draft data (binary data)

[0526] Specific operation: The server uses a natural language processing library (e.g., NLTK or spaCy) to analyze the text of the provided data.

[0527] Data processing: The server understands the context and structure of the document and extracts semantic elements (steps, tasks, important points, etc.).

[0528] Output: Based on the extracted information, the server generates flow data showing each step of the business flow and its sequence, and stores it in JSON format.

[0529] Step 3:

[0530] Receiving user change requests

[0531] Input: User instructions (text entered in natural language)

[0532] Specific operation: The user enters instructions into an interactive interface and presses the submit button.

[0533] Data processing: The terminal sends the entered text to the server.

[0534] Output: The server receives instructions (text) from the user.

[0535] Step 4:

[0536] Updating flow data

[0537] Input: User instruction text and current flow data (JSON format)

[0538] Specific operation: The server analyzes user instructions using natural language processing technology.

[0539] Data processing: Based on the instructions, perform operations such as adding, deleting, and modifying flow data (each step and its sequence).

[0540] Output: The server generates updated flow data (in JSON format).

[0541] Step 5:

[0542] Output generation and display

[0543] Input: Updated flow data (JSON format)

[0544] Specific operation: The server uses a graphical display library (e.g., D3.js or Graphviz) to generate new flowcharts and manuals.

[0545] Data processing: Convert flow data into a format that is easy to visualize (image file, PDF, etc.).

[0546] Output: The generated artifacts are sent to the terminal and displayed to the user.

[0547] Specific example

[0548] For example, if a user wants to create a project management flow, they upload a PDF document containing basic project steps as an initial draft. The server analyzes the document, extracts each step, and saves it as flow data in JSON format.

[0549] Next, the user enters "Add a task to Phase 2" into the interactive interface. The terminal sends this instruction to the server, which parses the instruction and updates the flow data. The updated flow diagram is then regenerated, sent to the terminal, and displayed to the user.

[0550] In this way, it becomes possible to automatically generate and update business flows and manuals based on preliminary documents, significantly reducing the effort required from users.

[0551] (Application Example 1)

[0552] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0553] In traditional factory operations, creating and updating work flows and operation manuals was a manual process requiring considerable effort and time. Furthermore, there was no system that could quickly generate work procedures based on initial drafts and update them in real time according to user instructions. This led to frequent inconsistencies in work flows and operational errors, making efficient work execution difficult. Additionally, there was a need for a system that could accurately understand and implement user instructions given in natural language.

[0554] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0555] In this invention, the server includes means for uploading draft data, means for analyzing the draft data to generate flow data, means for receiving instructions from a user, means for updating the flow data based on the instructions, means for generating data that reflects the updated flow data, and means for displaying the data to the user. The application installed on the industrial machine includes means for analyzing draft data uploaded by field workers using natural language processing technology to automatically generate business flows and operating procedures, and means for updating these business flows and operating procedures based on user instructions and graphically displaying the operation of the industrial machine and the business flow. This enables the automation of complex factory operations and operating procedures, and allows for the rapid generation and updating of business flows and manuals.

[0556] A "draft document" is a document used as an initial design for a certain workflow or procedure.

[0557] "Uploading" refers to the method of sending electronic files owned by a user to a server and importing them into the system.

[0558] "Methods for analyzing and generating flow data" refers to methods that analyze uploaded draft documents and automatically generate data on business flows and operating procedures.

[0559] "Means of receiving user instructions" refers to the methods by which the system accepts instructions that users input into it.

[0560] "Means for updating the flow data based on instructions" refers to a method of changing existing flow data to new content in accordance with instructions from the user.

[0561] "Means of generating documents that reflect updated flow data" refers to methods of creating documents that describe new business flows and operating procedures based on updated flow data.

[0562] "Means of displaying materials to users" refers to methods for showing users generated business flows and operating procedures.

[0563] "Applications installed on industrial machinery" refers to software that is integrated into robots and equipment used in factories to execute work flows and operating procedures.

[0564] "Natural language processing technology" refers to the technology that enables computers to understand the sentences and words that humans use in everyday life, and to automate the analysis and response processes.

[0565] "Draft documents uploaded by on-site workers" refers to initial design documents created by factory workers and sent to the system.

[0566] "Methods for automatically generating business flows and operating procedures" refers to methods in which a system creates business flows and operating procedures based on draft documents without manual intervention.

[0567] "Methods for updating business workflows and operating procedures" refers to methods of adding new information to the current business workflow or operating procedures, or modifying existing information, based on user instructions.

[0568] "Means of graphically displaying the operation of industrial machinery and business workflows" refers to methods of visually representing and presenting to users how to operate industrial machinery and the flow of work using diagrams and graphs.

[0569] This invention relates to a system that provides an application to be installed on industrial machinery used in a factory, and that automatically generates and updates business flows and operation manuals. This system performs a series of processes including uploading draft data, analysis, generation of flow data, receiving user instructions, updating flow data, and generating and displaying documents.

[0570] System Configuration

[0571] This system consists of the following main components:

[0572] 1. Method for uploading draft documents: Users upload draft documents (PDF, Word, Excel files, etc.) to the system. This upload function is provided through the user terminal interface.

[0573] 2. Means for analyzing draft data and generating flow data: The server analyzes the received draft data using natural language processing technology and generates flow data that shows each step of the business flow and its order. The NLP model used is "dbmdz / bert-large-cased-finetuned-conll03-english", which is part of the "transformers" library.

[0574] 3. Means of receiving user instructions: Users give instructions to the system through an interactive interface. These instructions are entered in natural language, making it usable even without special technical knowledge.

[0575] 4. Means for updating the flow data based on the instructions: The server analyzes the instructions from the user and updates the flow data accordingly. For example, if an instruction such as "Add a new step" is given, flow data reflecting the content and location of that instruction is generated.

[0576] 5. Means of generating documents that reflect updated flow data: Based on the updated flow data, new flowcharts and operation manuals are generated. Libraries and software for graphically drawing flowcharts are often used.

[0577] 6. Means for displaying the document to the user: The generated document is sent to the terminal and displayed to the user. The user can review the displayed document and issue additional modification instructions as needed.

[0578] Specific example

[0579] For example, when a factory worker sets up assembly procedures for a new module, they upload a PDF file containing the basic instructions. The server analyzes the PDF file, extracts each step, and generates flow data. The worker then enters an instruction into the interactive interface, such as "Add a new module inspection." The server analyzes this instruction and updates the flow data. The updated flow chart and work manual are regenerated and displayed to the worker. The worker reviews it and provides final approval or additional instructions as needed.

[0580] Example of a prompt

[0581] "We have uploaded the initial assembly instructions."

[0582] "Add a new step: Module inspection"

[0583] "Please remove the inspection step from the assembly procedure."

[0584] Thus, the present invention makes it easy to automatically generate and update work flows and operating procedures within a factory, significantly reducing the workload for users.

[0585] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0586] Step 1:

[0587] The user uploads draft documents. These documents can be in formats such as PDF, Word, or Excel. The terminal provides an interface for selecting and submitting these draft documents, and sends the selected files to the server. The input is the file uploaded by the user, and the output is the draft document sent to the server.

[0588] Step 2:

[0589] The server analyzes the received draft data. Natural language processing techniques are used for the analysis to understand the context and structure of the data. Specifically, the "dbmdz / bert-large-cased-finetuned-conll03-english" model from the "transformers" library is used to extract each step of the business flow from the draft data. The input is the received draft data, and the output is the generated flow data.

[0590] Step 3:

[0591] The server initially generates a business flow based on the generated flow data. Here, each step and its sequence are defined and visualized as a graphical flowchart. This flowchart is displayed to the user in subsequent steps. The input is the flow data, and the output is the visualized business flow diagram.

[0592] Step 4:

[0593] The user inputs instructions into the system through an interactive interface. These instructions are entered in natural language and include adding new steps, modifying or deleting existing ones, and more. The input consists of the user's instructions (prompts), and the output is the content of the received instructions.

[0594] Step 5:

[0595] The server analyzes the user's instructions and updates the flow data accordingly. For example, if the instruction is "Add a new step: module inspection," the flow data will be updated to reflect its content and location. The input is the user's instructions and the current flow data, and the output is the updated flow data.

[0596] Step 6:

[0597] The server generates new business process diagrams and operation manuals based on the updated flow data. New diagrams and procedures containing specific details are generated, and this information is visualized. The input is the updated flow data, and the output is the new business process diagrams and operation manuals.

[0598] Step 7:

[0599] The terminal displays the generated business process diagrams and operation manuals to the user. The user can review the displayed materials and issue instructions for further changes as needed. The input is the new business process diagrams and operation manuals, and the output is the visualized materials displayed to the user.

[0600] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0601] This invention relates to a system that automatically generates and updates business flows and manuals based on initial drafts created by the user, and further incorporates an emotion engine that recognizes the user's emotions to provide responses adapted to the user.

[0602] System Configuration

[0603] This system consists of the following main components:

[0604] 1. How to upload draft documents

[0605] 2. Means for analyzing draft data and generating flow data

[0606] 3. Means of receiving instructions from the user

[0607] 4. Means for updating flow data based on instructions

[0608] 5. Means for generating documents that reflect updated flow data

[0609] 6. Means of displaying materials to users

[0610] 7. Emotion engine that recognizes user emotions

[0611] 8. Means for adjusting the system's response based on recognized emotions.

[0612] The specific operation of the program

[0613] Upload of initial draft documents

[0614] The user selects and uploads their initial draft document (e.g., PDF, Word, or Excel file) through the system interface. The terminal then sends this file to the server. The sent file is temporarily stored on the server.

[0615] Data analysis and flow data generation

[0616] The server analyzes the received draft data. This analysis uses natural language processing techniques to understand the context and structure of the data (e.g., headings, paragraphs, lists). Based on the analysis results, flow data, which is a data model of the business process, is generated.

[0617] User change instructions

[0618] Users give instructions to the system through an interactive interface. This includes adding new steps, modifying or deleting existing steps, and more. Instructions are entered in natural language, so no special technical knowledge is required.

[0619] Use of the emotion engine

[0620] The emotion engine analyzes the user's voice and facial expressions to recognize their emotions. For example, it collects the user's voice tone and facial expressions through the camera and microphone when they input instructions, and analyzes their emotions. This helps determine whether the user is stressed or satisfied.

[0621] Flow data updates and response adjustments

[0622] The server integrates user instructions with the results of the emotion engine's analysis and updates the flow data accordingly. For example, if the user is stressed, the system displays more detailed guidance or additional support messages. Conversely, if the user is satisfied, it provides a concise response.

[0623] Output generation and display

[0624] Based on the updated flow data, new flow charts and manuals are generated. This process utilizes graphical drawing libraries and other tools. The generated materials and system responses are sent to the terminal and displayed to the user. The user can review the displayed materials and, if necessary, provide instructions for additional changes.

[0625] Specific example

[0626] For example, if a user wants to create a project management flow for the first time, they upload a document (e.g., a PDF) containing basic project steps as an initial draft. The server analyzes this document, extracts each step, and generates flow data.

[0627] Next, the user enters instructions into the dialogue interface, such as "Add a task to Phase 2." The server analyzes these instructions and updates the flow data accordingly. The emotion engine then analyzes the user's voice and facial expressions, and if it determines that the user is satisfied, it displays the updated flow diagram along with a relatively simple response. Conversely, if the user is stressed, the system provides more detailed explanations and guidance.

[0628] Thus, the present invention makes it possible to automatically generate and update business flows and manuals based on draft data, and furthermore, by recognizing user emotions and adjusting responses, it is possible to provide a more user-friendly system.

[0629] The following describes the processing flow.

[0630] Understood. The specific steps of the process are outlined below.

[0631] Step 1:

[0632] Users select and upload their initial draft documents (e.g., PDF, Word, or Excel files) through the system interface.

[0633] Step 2:

[0634] The terminal sends the draft data selected by the user to the server. The sent files are temporarily stored on the server.

[0635] Step 3:

[0636] The server analyzes the received data. Natural language processing techniques are used for the analysis to understand the context and structure of the data (e.g., headings, paragraphs, lists). Based on the analysis results, a data model (flow data) of the business process is generated.

[0637] Step 4:

[0638] The terminal displays the generated flow data to the user as a graphical flowchart. The user then reviews it.

[0639] Step 5:

[0640] Users give instructions to the system through an interactive interface, including adding new steps, modifying or deleting existing ones.

[0641] Step 6:

[0642] The emotion engine collects and analyzes the user's voice and facial expressions through input devices (microphone and camera). The emotion engine recognizes the user's emotional state (e.g., feeling stressed, feeling satisfied).

[0643] Step 7:

[0644] The device sends user instructions and emotion recognition results from the emotion engine to the server.

[0645] Step 8:

[0646] The server analyzes user instructions using natural language processing techniques and updates flow data accordingly. It also adjusts the system's response based on the results of the emotion engine (e.g., providing detailed guidance if the user is stressed).

[0647] Step 9:

[0648] The server generates new flowcharts and manuals based on the updated flow data. A graphical drawing library is used for this purpose.

[0649] Step 10:

[0650] Along with the newly generated flowcharts and manuals, responses tailored to the user's emotions are sent to the terminal. For example, a concise response if the user is satisfied, and a detailed guide if they are stressed.

[0651] Step 11:

[0652] The terminal displays the generated new flowcharts, manuals, and system responses to the user. The user can review the content and, if necessary, provide instructions for additional changes.

[0653] Step 12:

[0654] If the user provides additional instructions, steps 5 through 11 are repeated. Finally, upon user approval, the updated flowchart and manual are finalized.

[0655] This series of steps allows users to efficiently create and update workflows and manuals, and furthermore, the system's responses are adjusted based on sentiment recognition, resulting in a better user experience.

[0656] (Example 2)

[0657] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0658] Traditional automated systems for generating business flows and manuals have limitations in improving the user experience because they provide uniform processing responses without considering user emotions or stress levels. Furthermore, a lack of convenience was a problem, as users often needed technical knowledge to issue instructions. Additionally, the difficulty for users to intuitively understand the update status and results of flow data was also an issue.

[0659] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0660] In this invention, the server includes means for uploading draft data, means for analyzing the draft data to generate flow data, and means for receiving instructions from the user. This makes it possible to recognize the user's emotions and stress levels and adjust the system response accordingly. Specifically, by including the analysis of the user's voice and facial expressions and adjusting the response based on the recognized emotions, the flow data update process becomes more user-friendly and the user can easily operate it with an intuitive interface.

[0661] A "draft document" is a document file format used by a system as the basis for analysis and data generation.

[0662] "Flow data" refers to a data model of a business process flow generated based on information analyzed from preliminary documents.

[0663] "User instructions" refer to input given by the user to the system, including the addition of new steps and the modification or deletion of existing steps.

[0664] "Means of recognizing emotions" refers to the function of a system that analyzes the user's voice and facial expressions to identify the user's emotional state.

[0665] "Means for adjusting responses" refers to a function that dynamically applies response content, such as confirmation messages and guidelines from the system, based on the recognized emotions of the user.

[0666] "Means of displaying materials to the user" refers to an interface for clearly displaying generated and updated flow data and related materials on the user's terminal.

[0667] This invention relates to a system that automatically generates and updates business flows and manuals based on initial drafts created by the user, and further incorporates an emotion engine that recognizes the user's emotions to provide responses adapted to the user.

[0668] System Configuration

[0669] This system consists of the following main components:

[0670] 1. How to upload draft documents

[0671] 2. Means for analyzing draft data and generating flow data

[0672] 3. Means of receiving instructions from the user

[0673] 4. Means for updating flow data based on instructions

[0674] 5. A means of recognizing emotions by analyzing the user's voice and facial expressions.

[0675] 6. Means for adjusting the system's response based on recognized emotions.

[0676] 7. Means for generating documents that reflect updated flow data

[0677] 8. Means of displaying materials to users

[0678] Detailed explanation of the program's processing

[0679] Upload of initial draft documents

[0680] The user selects and uploads a draft document (e.g., PDF, Word, or Excel file) created initially through the terminal's interface. The terminal sends this file to the server. The server stores the uploaded file in a temporary storage area.

[0681] Data analysis and flow data generation

[0682] The server analyzes the received draft data. This analysis uses natural language processing techniques (e.g., SpaCy, NLTK) to understand the context and structure of the data (headings, paragraphs, lists, etc.). Based on the analysis results, flow data, which is a data model of the business process, is generated.

[0683] User change instructions

[0684] The user provides instructions to the system through an interactive interface. This includes adding new steps, modifying or deleting existing ones. Instructions are entered in natural language, so no special technical knowledge is required. The terminal receives the user's input and sends it to the server.

[0685] Use of the emotion engine

[0686] The server receives user voice and facial expression data from the terminal and recognizes the user's emotions using an emotion engine. The emotion engine uses voice analysis (e.g., speech-to-text services) and facial expression analysis (e.g., OpenCV).

[0687] Flow data updates and response adjustments

[0688] The server integrates user instructions and sentiment analysis results, updating flow data accordingly. If the user is experiencing stress, the system displays detailed guidance and additional support messages. Conversely, if the user is satisfied, it provides a concise response.

[0689] Output generation and display

[0690] Based on the updated flow data, new flowcharts and manuals are generated. This process utilizes graphical drawing libraries (e.g., D3.js). The generated materials and system responses are sent to the terminal and displayed to the user.

[0691] Specific example

[0692] For example, if a user wants to create a project management flow, they would follow these steps:

[0693] The user uploads a PDF document outlining basic project steps as an initial draft. The server analyzes the document, extracts each step, and generates flow data. Next, the user inputs instructions, such as "Add a task to Phase 2," into the interactive interface. The server analyzes these instructions and updates the flow data accordingly. Subsequently, the emotion engine analyzes the user's voice and facial expressions, and if it determines that the user is satisfied, it displays an updated flow chart with a relatively simple response. Conversely, if the user is stressed, the system provides more detailed explanations and guidance.

[0694] Examples of prompts for generative AI models

[0695] "As an initial step in project management, please generate the following flow data. If you wish to add tasks to each step of Phase 1 and Phase 2, please provide appropriate instructions. Also, please adjust support messages according to the user's sentiment."

[0696] The above describes embodiments for carrying out the present invention.

[0697] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0698] Step 1:

[0699] Input: Initial draft documents created by the user (PDF, Word, Excel files)

[0700] Specific operation: The user clicks the "Upload" button on the device interface. The device sends the selected file to the server as an HTTP POST request.

[0701] Output: The uploaded file is saved to the server.

[0702] Step 2:

[0703] Input: Uploaded draft document

[0704] Specific operation: The server reads the received file and converts it into string data. Natural language processing techniques (e.g., SpaCy or NLTK) are used to analyze the document's context and structure (headings, paragraphs, lists, etc.).

[0705] Output: Flow data (JSON format) generated based on the analysis results.

[0706] Step 3:

[0707] Input: Flow data analysis results

[0708] Specific operation: The server displays initial flow data in the user interface. The user inputs instructions such as adding, modifying, or deleting steps in natural language through an interactive interface.

[0709] Output: User instructions are sent to the server via the terminal.

[0710] Step 4:

[0711] Input: User instructions

[0712] Specific operation: The server receives user instructions and analyzes the content of the instructions using natural language understanding (NLU) technology (e.g., the BERT model). Based on the analyzed instructions, the server updates the flow data.

[0713] Output: Updated flow data

[0714] Step 5:

[0715] Input: User's voice and facial expression data

[0716] Specific operation: The device captures the user's voice and facial expressions through the microphone and camera and sends them to the server. The server converts the voice data into text using a speech-to-text service and recognizes emotions using facial expression analysis technology (e.g., OpenCV).

[0717] Output: Emotion recognition result

[0718] Step 6:

[0719] Input: Emotion recognition result

[0720] Specific operation: The server adjusts its response based on the emotion recognition results and user instructions. For example, if the user is stressed, it generates detailed guides or additional support messages; if the user is satisfied, it provides a concise response.

[0721] Output: Adjusted response

[0722] Step 7:

[0723] Input: Updated flow data

[0724] Specific operation: The server uses a graphical drawing library (e.g., D3.js) to generate new flowcharts and manuals based on the updated flow data.

[0725] Output: Generated flowcharts and manuals

[0726] Step 8:

[0727] Input: Generated flowcharts and manuals, and adjusted responses.

[0728] Specific operation: The server sends these deliverables to the terminal. The terminal displays the received materials and responses on the screen.

[0729] Output: Flowcharts, manuals, and system response messages displayed in a user-readable format.

[0730] The above describes the processing steps of this system.

[0731] (Application Example 2)

[0732] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0733] The creation and updating of conventional work flows and manuals is extremely time-consuming and difficult to address in a way that takes into account the emotions and stress levels of workers. As a result, work efficiency may decrease, and worker satisfaction may also decline. This problem is particularly serious in workplaces such as factories, where real-time flow updates and immediate feedback are required. This invention aims to solve these problems.

[0734] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for uploading draft data, means for analyzing the draft data to generate business flow data, means for receiving instructions from the user, means for updating the business flow data based on the instructions, means for recognizing the user's emotions using emotion recognition means, means for adjusting the response based on the recognized emotions, means for generating a document that reflects the updated business flow data, and means for displaying the document to the user. This enables the automatic and efficient generation and updating of business flows and manuals, as well as flexible responses that respond to the emotions of the workers.

[0735] A "draft document" is a document containing the basic content that was created initially.

[0736] "Business process flow data" refers to a data model that represents the flow and procedures of a business process.

[0737] "User instructions" refer to commands given by users to the system, such as additions, deletions, and modifications.

[0738] "Emotion recognition means" refers to a function that analyzes the user's voice and facial expressions to recognize their emotions.

[0739] "Means of adjusting responses" refers to a function that optimizes the system's response based on recognized emotions.

[0740] "Means for generating documents" refers to a function that automatically creates new documents based on analyzed and updated flow data.

[0741] "Means of displaying materials to users" refers to a function that displays generated materials in a format that users can review.

[0742] This invention relates to a system that automatically generates and updates business flow data based on a draft document created by the user initially. The system incorporates a function that recognizes the user's emotions and adjusts the response accordingly.

[0743] System Configuration

[0744] This system consists of the following main components:

[0745] 1. How to upload draft documents

[0746] 2. A means of generating business flow data by analyzing draft documents.

[0747] 3. Means of receiving instructions from the user

[0748] 4. Means for recognizing a user's emotions using emotion recognition means

[0749] 5. Means of adjusting responses based on perceived emotions

[0750] 6. Means for generating documents that reflect updated business flow data

[0751] 7. Means for displaying the aforementioned materials to the user

[0752] Specific hardware and software to be used

[0753] hardware

[0754] Factory robots: Responsible for generating and displaying work instructions and documents.

[0755] Camera: Used to capture the user's facial expressions and recognize emotions.

[0756] Microphone: Used to capture the user's voice and recognize emotions and instructions.

[0757] Tablets and monitors: Used to visually display updated workflow data.

[0758] software

[0759] Natural language processing libraries (e.g., SpaCy): Used to analyze uploaded draft documents and generate business flow data.

[0760] Emotion recognition engine (e.g., Microsoft Azure Face API, Google Cloud Speech-to-Text): Used to analyze and recognize the user's emotions.

[0761] Flow data generation and update libraries (e.g., Graphviz): Used to generate data models of business processes and to draw visual flow diagrams based on updated data.

[0762] Flow of operations

[0763] The user uploads a draft document (e.g., PDF, Word, Excel file) they initially created. This document is sent to the server by the system and temporarily stored. The server uses a natural language processing library to analyze the document and generate business flow data. Next, the user provides instructions through an interactive interface, which are entered in natural language. These instructions are then analyzed, and the business flow data is updated.

[0764] Furthermore, emotion recognition measures analyze the user's voice and facial expressions to recognize their emotions. Based on these results, the response is adjusted. For example, if the user is stressed, a detailed support message is provided. On the other hand, if the user is satisfied, a concise response is provided.

[0765] Based on updated workflow data, new documents are generated and displayed on tablets and monitors by factory robots.

[0766] Examples of specific cases and prompt statements

[0767] For example, consider a scenario where a factory worker uploads a work procedure manual they created for the first time. After uploading a PDF as a draft document to the system, the system analyzes it and generates business flow data. When the worker gives instructions in natural language, such as "Please add a new task to the next step," these instructions are analyzed and the business flow data is updated accordingly.

[0768] The emotion recognition system analyzes the worker's voice and facial expressions, and if the worker is experiencing stress, the system displays detailed guidance or additional support messages. Conversely, if the worker is satisfied, it provides a simple response.

[0769] The updated flowchart is displayed on a tablet or monitor, allowing workers to review it and provide additional instructions as needed.

[0770] Example of a prompt

[0771] "Please upload the work procedure manual."

[0772] "Add a new task to the next step."

[0773] "We are analyzing the emotions of the workers through cameras and microphones."

[0774] "We are adjusting our response based on the analysis results."

[0775] Thus, the present invention enables the automatic and efficient generation and updating of business flows and manuals, and also allows for flexible responses that respond to the emotions of the workers.

[0776] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0777] Step 1:

[0778] The user uploads a draft document (e.g., PDF, Word, Excel file) via their device. This draft document is sent to the server and temporarily stored. The input is the draft document in file format, and the output is the saved draft document file.

[0779] Step 2:

[0780] The server analyzes the uploaded draft data using a natural language processing library (e.g., SpaCy). The analysis generates business flow data. The input is the saved draft data file, and the output is the generated business flow data.

[0781] Specific operations include text extraction and contextual analysis of the draft materials.

[0782] Step 3:

[0783] The user inputs instructions in natural language through an interactive interface. These instructions include adding new steps, modifying existing steps, and deleting existing ones. The input is the user's instructions, and the output is the analysis result.

[0784] Step 4:

[0785] The server analyzes user instructions and updates the business flow data. Natural language processing is also used for this analysis. The input is the analyzed user instructions, and the output is the updated business flow data.

[0786] Specific actions include extracting instructions and modifying corresponding business flow data.

[0787] Step 5:

[0788] The server analyzes the user's voice and facial expressions using emotion recognition tools to recognize their emotions. This analysis uses emotion recognition engines (e.g., Microsoft Azure Face API, Google Cloud Speech-to-Text). The input is the user's voice and facial expression data, and the output is the recognized emotion.

[0789] Step 6:

[0790] The server adjusts its response based on the perceived emotion. Specifically, it provides a detailed support message if the user is stressed, and a concise response if they are satisfied. The input is the perceived emotion, and the output is the adjusted response.

[0791] Step 7:

[0792] The server generates new documents based on updated business process flow data and displays them on tablets and monitors via factory robots. A flow data generation and updating library (e.g., Graphviz) is used for generation. The input is the updated business process flow data, and the output is the generated documents (flowcharts and manuals).

[0793] Step 8:

[0794] The user reviews the displayed materials and provides additional instructions as needed. The input is the generated materials, and the output is the user's new instructions.

[0795] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0796] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0797] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0798] [Third Embodiment]

[0799] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0800] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0801] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0802] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0803] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0804] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0805] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0806] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0807] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0808] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0809] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0810] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0811] This invention relates to a system that automatically generates and updates business flows and manuals based on initial draft documents created by a user. This system performs a series of processes including analysis of the draft documents, generation of flow data, reception of user instructions, updating of flow data, and generation and display of deliverables.

[0812] System Configuration

[0813] This system consists of the following main components:

[0814] 1. How to upload draft documents

[0815] 2. Means for analyzing draft data and generating flow data

[0816] 3. Means of receiving instructions from the user

[0817] 4. Means for updating flow data based on instructions

[0818] 5. Means for generating documents that reflect updated flow data

[0819] 6. Means of displaying materials to users

[0820] The program's specific operation

[0821] Upload of initial draft documents

[0822] Users upload draft documents through the system interface. Various document formats are accepted, including PDF, Word, and Excel files. The terminal then sends the files selected by the user to the server.

[0823] Data analysis and flow data generation

[0824] The server analyzes the received draft data. This analysis uses natural language processing technology to understand the context and structure of the data. Based on the analysis results, flow data is generated that shows each step of the business process and its sequence.

[0825] User change instructions

[0826] Users give instructions to the system through an interactive interface. This includes adding new steps, modifying or deleting existing steps, and more. Instructions are entered in natural language, so no special technical knowledge is required.

[0827] Updating flow data

[0828] The server analyzes user instructions and updates the flow data accordingly. For example, if an instruction to add a new step is given, flow data reflecting its content and location will be generated.

[0829] Generation and display of deliverables

[0830] Based on the updated flow data, new flowcharts and manuals are generated. This process utilizes libraries for graphically drawing flowcharts. The generated documents are sent to the terminal and displayed to the user. The user can review the displayed documents and, if necessary, provide instructions for additional changes.

[0831] Specific example

[0832] For example, if a user wants to create a project management flow for the first time, they upload a document (e.g., a PDF) containing basic project steps as an initial draft. The server analyzes this document, extracts each step, and generates flow data.

[0833] Next, the user enters instructions into the interactive interface, such as "Add a task to Phase 2." The server analyzes these instructions and updates the flow data accordingly. The updated flow diagram is then regenerated and displayed to the user. The user reviews the displayed flow diagram and provides final approval or additional instructions as needed.

[0834] Thus, the present invention makes it easy to automatically generate and update business flows and manuals based on draft materials, significantly reducing the effort required from users.

[0835] The following describes the processing flow.

[0836] Understood. The specific processing steps of the program are described below.

[0837] Step 1:

[0838] Users select and upload their initial draft documents (e.g., PDF, Word, or Excel files) through the system interface.

[0839] Step 2:

[0840] The terminal sends the draft data selected by the user to the server. The sent files are temporarily stored on the server.

[0841] Step 3:

[0842] The server analyzes the received data. Natural language processing techniques are used for the analysis to understand the context and structure of the data (e.g., headings, paragraphs, lists).

[0843] Step 4:

[0844] The server generates flow data, which is a data model of the business process, based on the analysis results. This data includes the order and content of each step.

[0845] Step 5:

[0846] The terminal displays the generated flow data to the user as a graphical flowchart. The user reviews this and enters instructions for changes or additions into the interactive interface.

[0847] Step 6:

[0848] The user inputs instructions in natural language through a conversational interface. For example, they might input instructions such as, "Add a new task after step 3."

[0849] Step 7:

[0850] The terminal sends the user's instructions to the server. The server receives the instructions from the user and analyzes their content using natural language processing.

[0851] Step 8:

[0852] The server updates the flow data based on the instructions it has analyzed. For example, it might add new tasks or modify existing steps as instructed.

[0853] Step 9:

[0854] The server regenerates new flowcharts and manuals based on the updated flow data. A graphical drawing library is used for this purpose.

[0855] Step 10:

[0856] Updated flowcharts and manuals are sent from the server to the terminal. The terminal displays them, and the user reviews the content.

[0857] Step 11:

[0858] If the user requests additional changes, steps 6 through 10 are repeated. Once the user finally approves, the updated flowcharts and manuals are finalized.

[0859] This series of steps allows users to efficiently create and update workflows and manuals.

[0860] (Example 1)

[0861] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0862] To improve operational efficiency and reduce workload, it is necessary to automatically generate and update subsequent workflows and manuals based on the basic documents created by users initially. However, with current systems, users need to spend a lot of time and effort updating workflows and manuals, which does not contribute to operational efficiency. Furthermore, generating these often requires technical knowledge, and the need for specialized skills is a challenge.

[0863] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0864] In this invention, the server includes means for uploading draft data, means for analyzing the draft data and generating flow data, means for receiving instructions from the user, means for generating deliverables based on the updated flow data, and means for displaying the generated deliverables to the user. This enables the automatic generation and updating of business flows and manuals based on draft data.

[0865] A "draft document" is a basic document that serves as the basis for the user's initial workflow and manual.

[0866] "Flow data" refers to data generated by analyzing draft documents, which shows each step and its sequence in a business process.

[0867] "Analysis methods" refer to methods for analyzing preliminary data to understand its context and structure, and generating flow data.

[0868] "Generation method" refers to a means of generating deliverables such as new business flows and manuals based on updated flow data.

[0869] "Natural language processing technology" is a technology that uses computers to analyze and understand human natural language.

[0870] A "graphical display library" is a software library used to visually display business flows and data.

[0871] A "document file" refers to files that include business-related document formats such as PDF, Word, and Excel files.

[0872] A "conversational interface" is a user interface that allows users to input instructions using natural language.

[0873] This invention relates to a system that automatically generates and updates business flows and manuals based on initial draft documents created by the user. This system performs a series of processes including uploading the draft documents, analyzing the documents, generating flow data, receiving user instructions, updating the flow data, and generating and displaying deliverables.

[0874] System Configuration

[0875] This system consists of the following main components:

[0876] 1. How to upload draft documents

[0877] 2. Means for analyzing draft data and generating flow data

[0878] 3. Means of receiving instructions from the user

[0879] 4. Means for updating flow data based on the above instructions

[0880] 5. Means for generating deliverables based on updated flow data

[0881] 6. Means for displaying the generated deliverables to the user.

[0882] The program's specific operation

[0883] Upload of initial draft documents

[0884] Users upload draft documents through the system interface. Various document formats are accepted, including PDF, Word, and Excel files. The terminal then sends the files selected by the user to the server.

[0885] Data analysis and flow data generation

[0886] The server analyzes the received data. Here, Python-based natural language processing technologies (e.g., NLTK or spaCy) are used to understand the context and structure of the data. Based on the analysis results, flow data is generated that shows each step of the business process and its sequence. The flow data is saved in JSON format.

[0887] Receiving user change requests

[0888] The user gives instructions to the system using an interactive interface. This interface consists of text boxes and selection menus and accepts instructions entered in natural language. The terminal sends the user's instructions to the server.

[0889] Updating flow data

[0890] The server analyzes user instructions and updates flow data accordingly. It uses natural language processing technology to understand instructions and performs operations (addition, deletion, modification) on the relevant flow data.

[0891] Generation and display of deliverables

[0892] Based on the updated flow data, new flowcharts and manuals are generated. This uses libraries for graphically rendering flowcharts (e.g., D3.js or Graphviz). The generated materials are sent to the terminal and displayed to the user.

[0893] Specific example

[0894] For example, if a user wants to create a project management flow, they upload a PDF document outlining basic project steps as an initial draft. The server analyzes the document, extracts each step, and generates flow data.

[0895] Next, the user enters instructions into the interactive interface, such as "Add a task to Phase 2." The server analyzes these instructions and updates the flow data accordingly. The updated flow diagram is then regenerated and displayed to the user.

[0896] Prompt text examples

[0897] "Please analyze this PDF document and generate flow data."

[0898] "Please add a new task to Phase 2 of the flowchart."

[0899] "Please generate and display the updated flowchart."

[0900] In this way, the system of the present invention enables the automatic generation and updating of business flows and manuals based on draft data, significantly reducing the effort required from the user.

[0901] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0902] System program processing steps

[0903] Step 1:

[0904] Upload of initial draft documents

[0905] Input: The user selects draft documents (PDF, Word, Excel files, etc.) and uploads them to the system.

[0906] Specific action: The user clicks the file selection button on the interface and selects a document file. Then, they press the upload button.

[0907] Data processing: The terminal retrieves the binary data of the selected file and sends it to the server via an HTTP POST request.

[0908] Output: The server receives the uploaded file.

[0909] Step 2:

[0910] Analysis of preliminary data and generation of flow data

[0911] Input: Received draft data (binary data)

[0912] Specific operation: The server uses a natural language processing library (e.g., NLTK or spaCy) to analyze the text of the provided data.

[0913] Data processing: The server understands the context and structure of the document and extracts semantic elements (steps, tasks, important points, etc.).

[0914] Output: Based on the extracted information, the server generates flow data showing each step of the business flow and its sequence, and stores it in JSON format.

[0915] Step 3:

[0916] Receiving user change requests

[0917] Input: User instructions (text entered in natural language)

[0918] Specific operation: The user enters instructions into an interactive interface and presses the submit button.

[0919] Data processing: The terminal sends the entered text to the server.

[0920] Output: The server receives instructions (text) from the user.

[0921] Step 4:

[0922] Updating flow data

[0923] Input: User instruction text and current flow data (JSON format)

[0924] Specific operation: The server analyzes user instructions using natural language processing technology.

[0925] Data processing: Based on the instructions, perform operations such as adding, deleting, and modifying flow data (each step and its sequence).

[0926] Output: The server generates updated flow data (in JSON format).

[0927] Step 5:

[0928] Generation and display of deliverables

[0929] Input: Updated flow data (JSON format)

[0930] Specific operation: The server uses a graphical display library (e.g., D3.js or Graphviz) to generate new flowcharts and manuals.

[0931] Data processing: Convert flow data into a format that is easy to visualize (image file, PDF, etc.).

[0932] Output: The generated artifacts are sent to the terminal and displayed to the user.

[0933] Specific example

[0934] For example, if a user wants to create a project management flow, they upload a PDF document outlining the basic project steps as an initial draft. The server analyzes the document, extracts each step, and saves it as flow data in JSON format.

[0935] Next, the user enters "Add a task to Phase 2" into the interactive interface. The terminal sends this instruction to the server, which parses the instruction and updates the flow data. The updated flow diagram is then regenerated, sent to the terminal, and displayed to the user.

[0936] In this way, it becomes possible to automatically generate and update business flows and manuals based on preliminary documents, significantly reducing the effort required from users.

[0937] (Application Example 1)

[0938] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0939] In traditional factory operations, creating and updating work flows and operation manuals was a manual process requiring considerable effort and time. Furthermore, there was no system that could quickly generate work procedures based on initial drafts and update them in real time according to user instructions. This led to frequent inconsistencies in work flows and operational errors, making efficient work execution difficult. Additionally, there was a need for a system that could accurately understand and implement user instructions given in natural language.

[0940] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0941] In this invention, the server includes means for uploading draft data, means for analyzing the draft data to generate flow data, means for receiving instructions from a user, means for updating the flow data based on the instructions, means for generating data that reflects the updated flow data, and means for displaying the data to the user. The application installed on the industrial machine includes means for analyzing draft data uploaded by field workers using natural language processing technology to automatically generate business flows and operating procedures, and means for updating these business flows and operating procedures based on user instructions and graphically displaying the operation of the industrial machine and the business flow. This enables the automation of complex factory operations and operating procedures, and allows for the rapid generation and updating of business flows and manuals.

[0942] A "draft document" is a document used as an initial design for a certain workflow or procedure.

[0943] "Uploading" refers to the method of sending electronic files owned by a user to a server and importing them into the system.

[0944] "Methods for analyzing and generating flow data" refers to methods that analyze uploaded draft documents and automatically generate data on business flows and operating procedures.

[0945] "Means of receiving user instructions" refers to the methods by which the system accepts instructions that users input into it.

[0946] "Means for updating the flow data based on instructions" refers to a method of changing existing flow data to new content in accordance with instructions from the user.

[0947] "Means of generating documents that reflect updated flow data" refers to methods of creating documents that describe new business flows and operating procedures based on updated flow data.

[0948] "Means of displaying materials to users" refers to methods for showing users generated business flows and operating procedures.

[0949] "Applications installed on industrial machinery" refers to software that is integrated into robots and equipment used in factories to execute work flows and operating procedures.

[0950] "Natural language processing technology" refers to the technology that enables computers to understand the sentences and words that humans use in everyday life, and to automate the analysis and response processes.

[0951] "Draft documents uploaded by on-site workers" refers to initial design documents created by factory workers and sent to the system.

[0952] "Methods for automatically generating business flows and operating procedures" refers to methods in which a system creates business flows and operating procedures based on draft documents without manual intervention.

[0953] "Methods for updating business workflows and operating procedures" refers to methods of adding new information to the current business workflow or operating procedures, or modifying existing information, based on user instructions.

[0954] "Means of graphically displaying the operation of industrial machinery and business workflows" refers to methods of visually representing and presenting to users how to operate industrial machinery and the flow of work using diagrams and graphs.

[0955] This invention relates to a system that provides an application to be installed on industrial machinery used in a factory, and that automatically generates and updates business flows and operation manuals. This system performs a series of processes including uploading draft data, analysis, generation of flow data, receiving user instructions, updating flow data, and generating and displaying documents.

[0956] System Configuration

[0957] This system consists of the following main components:

[0958] 1. Method for uploading draft documents: Users upload draft documents (PDF, Word, Excel files, etc.) to the system. This upload function is provided through the user terminal interface.

[0959] 2. Means for analyzing draft data and generating flow data: The server analyzes the received draft data using natural language processing technology and generates flow data that shows each step of the business flow and its order. The NLP model used is "dbmdz / bert-large-cased-finetuned-conll03-english", which is part of the "transformers" library.

[0960] 3. Means of receiving user instructions: Users provide instructions to the system through an interactive interface. These instructions are entered in natural language, making it usable even without special technical knowledge.

[0961] 4. Means for updating the flow data based on the instructions: The server analyzes the instructions from the user and updates the flow data accordingly. For example, if an instruction such as "Add a new step" is given, flow data reflecting the content and location of that instruction is generated.

[0962] 5. Means of generating documents that reflect updated flow data: Based on the updated flow data, new flowcharts and operation manuals are generated. Libraries and software for graphically drawing flowcharts are often used.

[0963] 6. Means for displaying the document to the user: The generated document is sent to the terminal and displayed to the user. The user can review the displayed document and issue additional modification instructions as needed.

[0964] Specific example

[0965] For example, when a factory worker sets up assembly procedures for a new module, they upload a PDF file containing the basic instructions. The server analyzes the PDF file, extracts each step, and generates flow data. The worker then enters an instruction into the interactive interface, such as "Add a new module inspection." The server analyzes this instruction and updates the flow data. The updated flow chart and work manual are regenerated and displayed to the worker. The worker reviews it and provides final approval or additional instructions as needed.

[0966] Example of a prompt

[0967] "We have uploaded the initial assembly instructions."

[0968] "Add a new step: Module inspection"

[0969] "Please remove the inspection step from the assembly procedure."

[0970] Thus, the present invention makes it easy to automatically generate and update work flows and operating procedures within a factory, significantly reducing the workload for users.

[0971] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0972] Step 1:

[0973] The user uploads draft documents. These documents can be in formats such as PDF, Word, or Excel. The terminal provides an interface for selecting and submitting these draft documents, and sends the selected files to the server. The input is the file uploaded by the user, and the output is the draft document sent to the server.

[0974] Step 2:

[0975] The server analyzes the received draft data. Natural language processing techniques are used for the analysis to understand the context and structure of the data. Specifically, the "dbmdz / bert-large-cased-finetuned-conll03-english" model from the "transformers" library is used to extract each step of the business flow from the draft data. The input is the received draft data, and the output is the generated flow data.

[0976] Step 3:

[0977] The server initially generates a business flow based on the generated flow data. Here, each step and its sequence are defined and visualized as a graphical flowchart. This flowchart is displayed to the user in subsequent steps. The input is the flow data, and the output is the visualized business flow diagram.

[0978] Step 4:

[0979] The user inputs instructions into the system through an interactive interface. These instructions are entered in natural language and include adding new steps, modifying or deleting existing ones, and more. The input consists of the user's instructions (prompts), and the output is the content of the received instructions.

[0980] Step 5:

[0981] The server analyzes the user's instructions and updates the flow data accordingly. For example, if the instruction is "Add a new step: module inspection," the flow data will be updated to reflect its content and location. The input is the user's instructions and the current flow data, and the output is the updated flow data.

[0982] Step 6:

[0983] The server generates new business process diagrams and operation manuals based on the updated flow data. New diagrams and procedures containing specific details are generated, and this information is visualized. The input is the updated flow data, and the output is the new business process diagrams and operation manuals.

[0984] Step 7:

[0985] The terminal displays the generated business process diagrams and operation manuals to the user. The user can review the displayed materials and issue instructions for further changes as needed. The input is the new business process diagrams and operation manuals, and the output is the visualized materials displayed to the user.

[0986] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0987] This invention relates to a system that automatically generates and updates business flows and manuals based on initial drafts created by the user, and further incorporates an emotion engine that recognizes the user's emotions to provide responses adapted to the user.

[0988] System Configuration

[0989] This system consists of the following main components:

[0990] 1. How to upload draft documents

[0991] 2. Means for analyzing draft data and generating flow data

[0992] 3. Means of receiving instructions from the user

[0993] 4. Means for updating flow data based on instructions

[0994] 5. Means for generating documents that reflect updated flow data

[0995] 6. Means of displaying materials to users

[0996] 7. Emotion engine that recognizes user emotions

[0997] 8. Means for adjusting the system's response based on recognized emotions.

[0998] The program's specific operation

[0999] Upload of initial draft documents

[1000] The user selects and uploads their initial draft document (e.g., PDF, Word, or Excel file) through the system interface. The terminal then sends this file to the server. The sent file is temporarily stored on the server.

[1001] Data analysis and flow data generation

[1002] The server analyzes the received draft data. This analysis uses natural language processing techniques to understand the context and structure of the data (e.g., headings, paragraphs, lists). Based on the analysis results, flow data, which is a data model of the business process, is generated.

[1003] User change instructions

[1004] Users give instructions to the system through an interactive interface. This includes adding new steps, modifying or deleting existing steps, and more. Instructions are entered in natural language, so no special technical knowledge is required.

[1005] Use of the emotion engine

[1006] The emotion engine analyzes the user's voice and facial expressions to recognize their emotions. For example, it collects the user's voice tone and facial expressions through the camera and microphone when they input instructions, and analyzes their emotions. This helps determine whether the user is stressed or satisfied.

[1007] Flow data updates and response adjustments

[1008] The server integrates user instructions with the results of the emotion engine's analysis and updates the flow data accordingly. For example, if the user is stressed, the system displays more detailed guidance or additional support messages. Conversely, if the user is satisfied, it provides a concise response.

[1009] Generation and display of deliverables

[1010] Based on the updated flow data, new flow charts and manuals are generated. This process utilizes graphical drawing libraries and other tools. The generated materials and system responses are sent to the terminal and displayed to the user. The user can review the displayed materials and, if necessary, provide instructions for additional changes.

[1011] Specific example

[1012] For example, if a user wants to create a project management flow for the first time, they upload a document (e.g., a PDF) containing basic project steps as an initial draft. The server analyzes this document, extracts each step, and generates flow data.

[1013] Next, the user enters instructions into the dialogue interface, such as "Add a task to Phase 2." The server analyzes these instructions and updates the flow data accordingly. The emotion engine then analyzes the user's voice and facial expressions, and if it determines that the user is satisfied, it displays the updated flow diagram along with a relatively simple response. Conversely, if the user is stressed, the system provides more detailed explanations and guidance.

[1014] Thus, the present invention makes it possible to automatically generate and update business flows and manuals based on draft data, and furthermore, by recognizing user emotions and adjusting responses, it is possible to provide a more user-friendly system.

[1015] The following describes the processing flow.

[1016] Understood. The specific steps of the process are outlined below.

[1017] Step 1:

[1018] Users select and upload their initial draft documents (e.g., PDF, Word, or Excel files) through the system interface.

[1019] Step 2:

[1020] The terminal sends the draft data selected by the user to the server. The sent files are temporarily stored on the server.

[1021] Step 3:

[1022] The server analyzes the received data. Natural language processing techniques are used for the analysis to understand the context and structure of the data (e.g., headings, paragraphs, lists). Based on the analysis results, a data model (flow data) of the business process is generated.

[1023] Step 4:

[1024] The terminal displays the generated flow data to the user as a graphical flowchart. The user then reviews it.

[1025] Step 5:

[1026] Users give instructions to the system through an interactive interface, including adding new steps, modifying or deleting existing ones.

[1027] Step 6:

[1028] The emotion engine collects and analyzes the user's voice and facial expressions through input devices (microphone and camera). The emotion engine recognizes the user's emotional state (e.g., feeling stressed, feeling satisfied).

[1029] Step 7:

[1030] The device sends user instructions and emotion recognition results from the emotion engine to the server.

[1031] Step 8:

[1032] The server analyzes user instructions using natural language processing techniques and updates flow data accordingly. It also adjusts the system's response based on the results of the emotion engine (e.g., providing detailed guidance if the user is stressed).

[1033] Step 9:

[1034] The server generates new flowcharts and manuals based on the updated flow data. A graphical drawing library is used for this purpose.

[1035] Step 10:

[1036] Along with the newly generated flowcharts and manuals, responses tailored to the user's emotions are sent to the terminal. For example, a concise response if the user is satisfied, and a detailed guide if they are stressed.

[1037] Step 11:

[1038] The terminal displays the generated new flowcharts, manuals, and system responses to the user. The user can review the content and, if necessary, provide instructions for additional changes.

[1039] Step 12:

[1040] If the user provides additional instructions, steps 5 through 11 are repeated. Finally, once the user approves, the updated flowchart and manual are finalized.

[1041] This series of steps allows users to efficiently create and update workflows and manuals, and furthermore, the system's responses are adjusted based on sentiment recognition, resulting in a better user experience.

[1042] (Example 2)

[1043] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1044] Traditional automated systems for generating business flows and manuals have limited potential for improving the user experience because they provide uniform processing responses without considering user emotions or stress levels. Furthermore, a lack of convenience was a problem, as users often needed technical knowledge to issue instructions. Additionally, the difficulty for users to intuitively understand the update status and results of flow data was also an issue.

[1045] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1046] In this invention, the server includes means for uploading draft data, means for analyzing the draft data to generate flow data, and means for receiving instructions from the user. This makes it possible to recognize the user's emotions and stress levels and adjust the system response accordingly. Specifically, by including the analysis of the user's voice and facial expressions and adjusting the response based on the recognized emotions, the flow data update process becomes more user-friendly and the user can easily operate it with an intuitive interface.

[1047] A "draft document" is a document file format used by a system as the basis for analysis and data generation.

[1048] "Flow data" refers to a data model of a business process flow generated based on information analyzed from preliminary documents.

[1049] "User instructions" refer to input given by the user to the system, including the addition of new steps and the modification or deletion of existing steps.

[1050] "Means of recognizing emotions" refers to the function of a system that analyzes the user's voice and facial expressions to identify the user's emotional state.

[1051] "Means for adjusting responses" refers to a function that dynamically applies response content, such as confirmation messages and guidelines from the system, based on the recognized emotions of the user.

[1052] "Means of displaying materials to the user" refers to an interface for clearly displaying generated and updated flow data and related materials on the user's terminal.

[1053] This invention relates to a system that automatically generates and updates business flows and manuals based on initial drafts created by the user, and further incorporates an emotion engine that recognizes the user's emotions to provide responses adapted to the user.

[1054] System Configuration

[1055] This system consists of the following main components:

[1056] 1. How to upload draft documents

[1057] 2. Means for analyzing draft data and generating flow data

[1058] 3. Means of receiving instructions from the user

[1059] 4. Means for updating flow data based on instructions

[1060] 5. A means of recognizing emotions by analyzing the user's voice and facial expressions.

[1061] 6. Means for adjusting the system's response based on recognized emotions.

[1062] 7. Means for generating documents that reflect updated flow data

[1063] 8. Means for displaying materials to users

[1064] Detailed explanation of the program's processing

[1065] Upload of initial draft documents

[1066] The user selects and uploads a draft document (e.g., PDF, Word, or Excel file) created initially through the terminal's interface. The terminal sends this file to the server. The server stores the uploaded file in a temporary storage area.

[1067] Data analysis and flow data generation

[1068] The server analyzes the received draft data. This analysis uses natural language processing techniques (e.g., SpaCy, NLTK) to understand the context and structure of the data (headings, paragraphs, lists, etc.). Based on the analysis results, flow data, which is a data model of the business process, is generated.

[1069] User change instructions

[1070] The user provides instructions to the system through an interactive interface. This includes adding new steps, modifying or deleting existing ones. Instructions are entered in natural language, so no special technical knowledge is required. The terminal receives the user's input and sends it to the server.

[1071] Use of the emotion engine

[1072] The server receives user voice and facial expression data from the terminal and recognizes the user's emotions using an emotion engine. The emotion engine uses voice analysis (e.g., speech-to-text services) and facial expression analysis (e.g., OpenCV).

[1073] Flow data updates and response adjustments

[1074] The server integrates user instructions and sentiment analysis results, updating flow data accordingly. If the user is experiencing stress, the system displays detailed guidance and additional support messages. Conversely, if the user is satisfied, it provides a concise response.

[1075] Generation and display of deliverables

[1076] Based on the updated flow data, new flowcharts and manuals are generated. This process utilizes graphical drawing libraries (e.g., D3.js). The generated materials and system responses are sent to the terminal and displayed to the user.

[1077] Specific example

[1078] For example, if a user wants to create a project management flow, they would follow these steps:

[1079] The user uploads a PDF document outlining basic project steps as an initial draft. The server analyzes the document, extracts each step, and generates flow data. Next, the user inputs instructions, such as "Add a task to Phase 2," into the interactive interface. The server analyzes these instructions and updates the flow data accordingly. Subsequently, the emotion engine analyzes the user's voice and facial expressions, and if it determines that the user is satisfied, it displays an updated flow chart with a relatively simple response. Conversely, if the user is stressed, the system provides more detailed explanations and guidance.

[1080] Examples of prompts for generative AI models

[1081] "As an initial step in project management, please generate the following flow data. If you wish to add tasks to each step of Phase 1 and Phase 2, please provide appropriate instructions. Also, please adjust support messages according to the user's sentiment."

[1082] The above describes embodiments for carrying out the present invention.

[1083] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1084] Step 1:

[1085] Input: Initial draft documents created by the user (PDF, Word, Excel files)

[1086] Specific operation: The user clicks the "Upload" button on the device interface. The device sends the selected file to the server as an HTTP POST request.

[1087] Output: The uploaded file is saved to the server.

[1088] Step 2:

[1089] Input: Uploaded draft document

[1090] Specific operation: The server reads the received file and converts it into string data. Natural language processing techniques (e.g., SpaCy or NLTK) are used to analyze the document's context and structure (headings, paragraphs, lists, etc.).

[1091] Output: Flow data (JSON format) generated based on the analysis results.

[1092] Step 3:

[1093] Input: Flow data analysis results

[1094] Specific operation: The server displays initial flow data in the user interface. The user inputs instructions such as adding, modifying, or deleting steps in natural language through an interactive interface.

[1095] Output: User instructions are sent to the server via the terminal.

[1096] Step 4:

[1097] Input: User instructions

[1098] Specific operation: The server receives user instructions and analyzes the content of the instructions using natural language understanding (NLU) technology (e.g., the BERT model). Based on the analyzed instructions, the server updates the flow data.

[1099] Output: Updated flow data

[1100] Step 5:

[1101] Input: User voice and facial expression data

[1102] Specific operation: The device captures the user's voice and facial expressions through the microphone and camera and sends them to the server. The server converts the voice data into text using a speech-to-text service and recognizes emotions using facial expression analysis technology (e.g., OpenCV).

[1103] Output: Emotion recognition result

[1104] Step 6:

[1105] Input: Emotion recognition result

[1106] Specific operation: The server adjusts its response based on the emotion recognition results and user instructions. For example, if the user is stressed, it generates detailed guides or additional support messages; if the user is satisfied, it provides a concise response.

[1107] Output: Adjusted response

[1108] Step 7:

[1109] Input: Updated flow data

[1110] Specific operation: The server uses a graphical drawing library (e.g., D3.js) to generate new flowcharts and manuals based on the updated flow data.

[1111] Output: Generated flowcharts and manuals

[1112] Step 8:

[1113] Input: Generated flowcharts and manuals, and adjusted responses.

[1114] Specific operation: The server sends these deliverables to the terminal. The terminal displays the received materials and responses on the screen.

[1115] Output: Flowcharts, manuals, and system response messages displayed in a user-readable format.

[1116] The above describes the processing steps of this system.

[1117] (Application Example 2)

[1118] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1119] The creation and updating of conventional work flows and manuals is extremely time-consuming and difficult to address in a way that takes into account the emotions and stress levels of workers. As a result, work efficiency may decrease, and worker satisfaction may also decline. This problem is particularly serious in workplaces such as factories, where real-time flow updates and immediate feedback are required. This invention aims to solve these problems.

[1120] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for uploading draft data, means for analyzing the draft data to generate business flow data, means for receiving instructions from the user, means for updating the business flow data based on the instructions, means for recognizing the user's emotions using emotion recognition means, means for adjusting the response based on the recognized emotions, means for generating a document that reflects the updated business flow data, and means for displaying the document to the user. This enables the automatic and efficient generation and updating of business flows and manuals, as well as flexible responses that respond to the emotions of the workers.

[1121] A "draft document" is a document containing the basic content that was created initially.

[1122] "Business process flow data" refers to a data model that represents the flow and procedures of a business process.

[1123] "User instructions" refer to commands given by the user to the system, such as additions, deletions, and modifications.

[1124] "Emotion recognition means" refers to a function that analyzes the user's voice and facial expressions to recognize their emotions.

[1125] "Means of adjusting responses" refers to a function that optimizes the system's response based on recognized emotions.

[1126] "Means for generating documents" refers to a function that automatically creates new documents based on analyzed and updated flow data.

[1127] "Means of displaying materials to users" refers to a function that displays generated materials in a format that users can review.

[1128] This invention relates to a system that automatically generates and updates business flow data based on a draft document created by the user initially. The system incorporates a function that recognizes the user's emotions and adjusts the response accordingly.

[1129] System Configuration

[1130] This system consists of the following main components:

[1131] 1. How to upload draft documents

[1132] 2. A means of generating business flow data by analyzing draft documents.

[1133] 3. Means of receiving instructions from the user

[1134] 4. Means for recognizing a user's emotions using emotion recognition means

[1135] 5. Means of adjusting responses based on perceived emotions

[1136] 6. Means for generating documents that reflect updated business flow data

[1137] 7. Means for displaying the aforementioned materials to the user

[1138] Specific hardware and software to be used

[1139] hardware

[1140] Factory robots: Responsible for generating and displaying work instructions and documents.

[1141] Camera: Used to capture the user's facial expressions and recognize emotions.

[1142] Microphone: Used to capture the user's voice and recognize emotions and instructions.

[1143] Tablets and monitors: Used to visually display updated workflow data.

[1144] software

[1145] Natural language processing libraries (e.g., SpaCy): Used to analyze uploaded draft documents and generate business flow data.

[1146] Emotion recognition engine (e.g., Microsoft Azure Face API, Google Cloud Speech-to-Text): Used to analyze and recognize the user's emotions.

[1147] Flow data generation and update libraries (e.g., Graphviz): Used to generate data models of business processes and to draw visual flow diagrams based on updated data.

[1148] Flow of operations

[1149] The user uploads a draft document (e.g., PDF, Word, Excel file) they initially created. This document is sent to the server by the system and temporarily stored. The server uses a natural language processing library to analyze the document and generate business flow data. Next, the user provides instructions through an interactive interface, which are entered in natural language. These instructions are then analyzed, and the business flow data is updated.

[1150] Furthermore, emotion recognition measures analyze the user's voice and facial expressions to recognize their emotions. Based on these results, the response is adjusted. For example, if the user is feeling stressed, a detailed support message is provided. On the other hand, if the user is satisfied, a concise response is provided.

[1151] Based on updated workflow data, new documents are generated and displayed on tablets and monitors by factory robots.

[1152] Examples of specific cases and prompt statements

[1153] For example, consider a scenario where a factory worker uploads a work procedure manual they created for the first time. After uploading a PDF as a draft document to the system, the system analyzes it and generates business flow data. When the worker gives instructions in natural language, such as "Please add a new task to the next step," these instructions are analyzed and the business flow data is updated accordingly.

[1154] The emotion recognition system analyzes the worker's voice and facial expressions, and if the worker is experiencing stress, the system displays detailed guidance or additional support messages. Conversely, if the worker is satisfied, it provides a simple response.

[1155] The updated flowchart is displayed on a tablet or monitor, allowing workers to review it and provide additional instructions as needed.

[1156] Example of a prompt

[1157] "Please upload the work procedure manual."

[1158] "Add a new task to the next step."

[1159] "We are analyzing the emotions of the workers through cameras and microphones."

[1160] "We are adjusting our response based on the analysis results."

[1161] Thus, the present invention enables the automatic and efficient generation and updating of business flows and manuals, and also allows for flexible responses that respond to the emotions of the workers.

[1162] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1163] Step 1:

[1164] The user uploads draft documents (e.g., PDF, Word, Excel files) via their device. These draft documents are sent to the server and temporarily stored. The input is the draft document in file format, and the output is the saved draft document file.

[1165] Step 2:

[1166] The server analyzes the uploaded draft data using a natural language processing library (e.g., SpaCy). The analysis generates business flow data. The input is the saved draft data file, and the output is the generated business flow data.

[1167] Specific operations include text extraction and contextual analysis of the draft materials.

[1168] Step 3:

[1169] The user inputs instructions in natural language through an interactive interface. These instructions include adding new steps, modifying existing steps, and deleting existing ones. The input is the user's instructions, and the output is the analysis result.

[1170] Step 4:

[1171] The server analyzes user instructions and updates the business flow data. Natural language processing is also used for this analysis. The input is the analyzed user instructions, and the output is the updated business flow data.

[1172] Specific actions include extracting instructions and modifying corresponding business flow data.

[1173] Step 5:

[1174] The server analyzes the user's voice and facial expressions using emotion recognition tools to recognize their emotions. This analysis uses emotion recognition engines (e.g., Microsoft Azure Face API, Google Cloud Speech-to-Text). The input is the user's voice and facial expression data, and the output is the recognized emotion.

[1175] Step 6:

[1176] The server adjusts its response based on the perceived emotion. Specifically, it provides a detailed support message if the user is stressed, and a concise response if they are satisfied. The input is the perceived emotion, and the output is the adjusted response.

[1177] Step 7:

[1178] The server generates new documents based on updated business process flow data and displays them on tablets and monitors via factory robots. A flow data generation and updating library (e.g., Graphviz) is used for generation. The input is the updated business process flow data, and the output is the generated documents (flowcharts and manuals).

[1179] Step 8:

[1180] The user reviews the displayed materials and provides additional instructions as needed. The input is the generated materials, and the output is the user's new instructions.

[1181] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1182] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1183] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1184] [Fourth Embodiment]

[1185] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1186] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1187] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1188] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1189] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1190] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1191] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1192] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1193] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1194] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1195] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1196] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1197] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1198] This invention relates to a system that automatically generates and updates business flows and manuals based on initial draft documents created by a user. This system performs a series of processes including analysis of the draft documents, generation of flow data, reception of user instructions, updating of flow data, and generation and display of deliverables.

[1199] System Configuration

[1200] This system consists of the following main components:

[1201] 1. How to upload draft documents

[1202] 2. Means for analyzing draft data and generating flow data

[1203] 3. Means of receiving instructions from the user

[1204] 4. Means for updating flow data based on instructions

[1205] 5. Means for generating documents that reflect updated flow data

[1206] 6. Means of displaying materials to users

[1207] The program's specific operation

[1208] Upload of initial draft documents

[1209] Users upload draft documents through the system interface. Various document formats are accepted, including PDF, Word, and Excel files. The terminal then sends the files selected by the user to the server.

[1210] Data analysis and flow data generation

[1211] The server analyzes the received draft data. This analysis uses natural language processing technology to understand the context and structure of the data. Based on the analysis results, flow data is generated that shows each step of the business process and its sequence.

[1212] User change instructions

[1213] Users give instructions to the system through an interactive interface. This includes adding new steps, modifying or deleting existing steps, and more. Instructions are entered in natural language, so no special technical knowledge is required.

[1214] Updating flow data

[1215] The server analyzes user instructions and updates the flow data accordingly. For example, if an instruction to add a new step is given, flow data reflecting its content and location will be generated.

[1216] Generation and display of deliverables

[1217] Based on the updated flow data, new flowcharts and manuals are generated. This process utilizes libraries for graphically drawing flowcharts. The generated documents are sent to the terminal and displayed to the user. The user can review the displayed documents and, if necessary, provide instructions for additional changes.

[1218] Specific example

[1219] For example, if a user wants to create a project management flow for the first time, they upload a document (e.g., a PDF) containing basic project steps as an initial draft. The server analyzes this document, extracts each step, and generates flow data.

[1220] Next, the user enters instructions into the interactive interface, such as "Add a task to Phase 2." The server analyzes these instructions and updates the flow data accordingly. The updated flow diagram is then regenerated and displayed to the user. The user reviews the displayed flow diagram and provides final approval or additional instructions as needed.

[1221] Thus, the present invention makes it easy to automatically generate and update business flows and manuals based on draft materials, significantly reducing the effort required from users.

[1222] The following describes the processing flow.

[1223] Understood. The specific processing steps of the program are described below.

[1224] Step 1:

[1225] Users select and upload their initial draft documents (e.g., PDF, Word, or Excel files) through the system interface.

[1226] Step 2:

[1227] The terminal sends the draft data selected by the user to the server. The sent files are temporarily stored on the server.

[1228] Step 3:

[1229] The server analyzes the received data. Natural language processing techniques are used for the analysis to understand the context and structure of the data (e.g., headings, paragraphs, lists).

[1230] Step 4:

[1231] The server generates flow data, which is a data model of the business process, based on the analysis results. This data includes the order and content of each step.

[1232] Step 5:

[1233] The terminal displays the generated flow data to the user as a graphical flowchart. The user reviews this and enters instructions for changes or additions into the interactive interface.

[1234] Step 6:

[1235] The user inputs instructions in natural language through a conversational interface. For example, they might input instructions such as, "Add a new task after step 3."

[1236] Step 7:

[1237] The terminal sends the user's instructions to the server. The server receives the instructions from the user and analyzes their content using natural language processing.

[1238] Step 8:

[1239] The server updates the flow data based on the instructions it has analyzed. For example, it might add new tasks or modify existing steps as instructed.

[1240] Step 9:

[1241] The server regenerates new flowcharts and manuals based on the updated flow data. A graphical drawing library is used for this purpose.

[1242] Step 10:

[1243] Updated flowcharts and manuals are sent from the server to the terminal. The terminal displays them, and the user reviews the content.

[1244] Step 11:

[1245] If the user requests additional changes, steps 6 through 10 are repeated. Once the user finally approves, the updated flowcharts and manuals are finalized.

[1246] This series of steps allows users to efficiently create and update workflows and manuals.

[1247] (Example 1)

[1248] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1249] To improve operational efficiency and reduce workload, it is necessary to automatically generate and update subsequent workflows and manuals based on the basic documents created by users initially. However, with current systems, users need to spend a lot of time and effort updating workflows and manuals, which does not contribute to operational efficiency. Furthermore, generating these often requires technical knowledge, and the need for specialized skills is a challenge.

[1250] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1251] In this invention, the server includes means for uploading draft data, means for analyzing the draft data and generating flow data, means for receiving instructions from the user, means for generating deliverables based on the updated flow data, and means for displaying the generated deliverables to the user. This enables the automatic generation and updating of business flows and manuals based on draft data.

[1252] A "draft document" is a basic document that serves as the basis for the user's initial workflow and manual.

[1253] "Flow data" refers to data generated by analyzing draft documents, which shows each step and its sequence in a business process.

[1254] "Analysis methods" refer to methods for analyzing preliminary data to understand its context and structure, and generating flow data.

[1255] "Generation method" refers to a means of generating deliverables such as new business flows and manuals based on updated flow data.

[1256] "Natural language processing technology" is a technology that uses computers to analyze and understand human natural language.

[1257] A "graphical display library" is a software library used to visually display business flows and data.

[1258] A "document file" refers to files that include business-related document formats such as PDF, Word, and Excel files.

[1259] A "conversational interface" is a user interface that allows users to input instructions using natural language.

[1260] This invention relates to a system that automatically generates and updates business flows and manuals based on initial draft documents created by the user. This system performs a series of processes including uploading the draft documents, analyzing the documents, generating flow data, receiving user instructions, updating the flow data, and generating and displaying deliverables.

[1261] System Configuration

[1262] This system consists of the following main components:

[1263] 1. How to upload draft documents

[1264] 2. Means for analyzing draft data and generating flow data

[1265] 3. Means of receiving instructions from the user

[1266] 4. Means for updating flow data based on the above instructions

[1267] 5. Means for generating deliverables based on updated flow data

[1268] 6. Means for displaying the generated deliverables to the user.

[1269] The program's specific operation

[1270] Upload of initial draft documents

[1271] Users upload draft documents through the system interface. Various document formats are accepted, including PDF, Word, and Excel files. The terminal then sends the files selected by the user to the server.

[1272] Data analysis and flow data generation

[1273] The server analyzes the received data. Here, Python-based natural language processing technologies (e.g., NLTK or spaCy) are used to understand the context and structure of the data. Based on the analysis results, flow data is generated that shows each step of the business process and its sequence. The flow data is saved in JSON format.

[1274] Receiving user change requests

[1275] The user gives instructions to the system using an interactive interface. This interface consists of text boxes and selection menus and accepts instructions entered in natural language. The terminal sends the user's instructions to the server.

[1276] Updating flow data

[1277] The server analyzes user instructions and updates flow data accordingly. It uses natural language processing technology to understand instructions and performs operations (addition, deletion, modification) on the relevant flow data.

[1278] Generation and display of deliverables

[1279] Based on the updated flow data, new flowcharts and manuals are generated. This uses libraries for graphically rendering flowcharts (e.g., D3.js or Graphviz). The generated materials are sent to the terminal and displayed to the user.

[1280] Specific example

[1281] For example, if a user wants to create a project management flow, they upload a PDF document outlining basic project steps as an initial draft. The server analyzes the document, extracts each step, and generates flow data.

[1282] Next, the user enters instructions into the interactive interface, such as "Add a task to Phase 2." The server analyzes these instructions and updates the flow data accordingly. The updated flow diagram is then regenerated and displayed to the user.

[1283] Prompt text examples

[1284] "Please analyze this PDF document and generate flow data."

[1285] "Please add a new task to Phase 2 of the flowchart."

[1286] "Please generate and display the updated flowchart."

[1287] In this way, the system of the present invention enables the automatic generation and updating of business flows and manuals based on draft data, significantly reducing the effort required from the user.

[1288] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1289] System program processing steps

[1290] Step 1:

[1291] Upload of initial draft documents

[1292] Input: The user selects draft documents (PDF, Word, Excel files, etc.) and uploads them to the system.

[1293] Specific action: The user clicks the file selection button on the interface and selects a document file. Then, they press the upload button.

[1294] Data processing: The terminal retrieves the binary data of the selected file and sends it to the server via an HTTP POST request.

[1295] Output: The server receives the uploaded file.

[1296] Step 2:

[1297] Analysis of preliminary data and generation of flow data

[1298] Input: Received draft data (binary data)

[1299] Specific operation: The server uses a natural language processing library (e.g., NLTK or spaCy) to analyze the text of the provided data.

[1300] Data processing: The server understands the context and structure of the document and extracts semantic elements (steps, tasks, important points, etc.).

[1301] Output: Based on the extracted information, the server generates flow data showing each step of the business flow and its sequence, and stores it in JSON format.

[1302] Step 3:

[1303] Receiving user change requests

[1304] Input: User instructions (text entered in natural language)

[1305] Specific operation: The user enters instructions into an interactive interface and presses the submit button.

[1306] Data processing: The terminal sends the entered text to the server.

[1307] Output: The server receives instructions (text) from the user.

[1308] Step 4:

[1309] Updating flow data

[1310] Input: User instruction text and current flow data (JSON format)

[1311] Specific operation: The server analyzes user instructions using natural language processing technology.

[1312] Data processing: Based on the instructions, perform operations such as adding, deleting, and modifying flow data (each step and its sequence).

[1313] Output: The server generates updated flow data (in JSON format).

[1314] Step 5:

[1315] Generation and display of deliverables

[1316] Input: Updated flow data (JSON format)

[1317] Specific operation: The server uses a graphical display library (e.g., D3.js or Graphviz) to generate new flowcharts and manuals.

[1318] Data processing: Convert flow data into a format that is easy to visualize (image file, PDF, etc.).

[1319] Output: The generated artifacts are sent to the terminal and displayed to the user.

[1320] Specific example

[1321] For example, if a user wants to create a project management flow, they upload a PDF document outlining the basic project steps as an initial draft. The server analyzes the document, extracts each step, and saves it as flow data in JSON format.

[1322] Next, the user enters "Add a task to Phase 2" into the interactive interface. The terminal sends this instruction to the server, which parses the instruction and updates the flow data. The updated flow diagram is then regenerated, sent to the terminal, and displayed to the user.

[1323] In this way, it becomes possible to automatically generate and update business flows and manuals based on preliminary documents, significantly reducing the effort required from users.

[1324] (Application Example 1)

[1325] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1326] In traditional factory operations, creating and updating work flows and operation manuals was a manual process requiring considerable effort and time. Furthermore, there was no system that could quickly generate work procedures based on initial drafts and update them in real time according to user instructions. This led to frequent inconsistencies in work flows and operational errors, making efficient work execution difficult. Additionally, there was a need for a system that could accurately understand and implement user instructions given in natural language.

[1327] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1328] In this invention, the server includes means for uploading draft data, means for analyzing the draft data to generate flow data, means for receiving instructions from a user, means for updating the flow data based on the instructions, means for generating data that reflects the updated flow data, and means for displaying the data to the user. The application installed on the industrial machine includes means for analyzing draft data uploaded by field workers using natural language processing technology to automatically generate business flows and operating procedures, and means for updating these business flows and operating procedures based on user instructions and graphically displaying the operation of the industrial machine and the business flow. This enables the automation of complex factory operations and operating procedures, and allows for the rapid generation and updating of business flows and manuals.

[1329] A "draft document" is a document used as an initial design for a certain workflow or procedure.

[1330] "Uploading" refers to the method of sending electronic files owned by a user to a server and importing them into the system.

[1331] "Methods for analyzing and generating flow data" refers to methods that analyze uploaded draft documents and automatically generate data on business flows and operating procedures.

[1332] "Means of receiving user instructions" refers to the methods by which the system accepts instructions that users input into it.

[1333] "Means for updating the flow data based on instructions" refers to a method of changing existing flow data to new content in accordance with instructions from the user.

[1334] "Means of generating documents that reflect updated flow data" refers to methods of creating documents that describe new business flows and operating procedures based on updated flow data.

[1335] "Means of displaying materials to users" refers to methods for showing users generated business flows and operating procedures.

[1336] "Applications installed on industrial machinery" refers to software that is integrated into robots and equipment used in factories to execute work flows and operating procedures.

[1337] "Natural language processing technology" refers to the technology that enables computers to understand the sentences and words that humans use in everyday life, and to automate the analysis and response processes.

[1338] "Draft documents uploaded by on-site workers" refers to initial design documents created by factory workers and sent to the system.

[1339] "Methods for automatically generating business flows and operating procedures" refers to methods in which a system creates business flows and operating procedures based on draft documents without manual intervention.

[1340] "Methods for updating business workflows and operating procedures" refers to methods of adding new information to the current business workflow or operating procedures, or modifying existing information, based on user instructions.

[1341] "Means of graphically displaying the operation of industrial machinery and business workflows" refers to methods of visually representing and presenting to users how to operate industrial machinery and the flow of work using diagrams and graphs.

[1342] This invention relates to a system that provides an application to be installed on industrial machinery used in a factory, and that automatically generates and updates business flows and operation manuals. This system performs a series of processes including uploading draft data, analysis, generation of flow data, receiving user instructions, updating flow data, and generating and displaying documents.

[1343] System Configuration

[1344] This system consists of the following main components:

[1345] 1. Method for uploading draft documents: Users upload draft documents (PDF, Word, Excel files, etc.) to the system. This upload function is provided through the user terminal interface.

[1346] 2. Means for analyzing draft data and generating flow data: The server analyzes the received draft data using natural language processing technology and generates flow data that shows each step of the business flow and its order. The NLP model used is "dbmdz / bert-large-cased-finetuned-conll03-english", which is part of the "transformers" library.

[1347] 3. Means of receiving user instructions: Users provide instructions to the system through an interactive interface. These instructions are entered in natural language, making it usable even without special technical knowledge.

[1348] 4. Means for updating the flow data based on the instructions: The server analyzes the instructions from the user and updates the flow data accordingly. For example, if an instruction such as "Add a new step" is given, flow data reflecting the content and location of that instruction is generated.

[1349] 5. Means of generating documents that reflect updated flow data: Based on the updated flow data, new flowcharts and operation manuals are generated. Libraries and software for graphically drawing flowcharts are often used.

[1350] 6. Means for displaying the document to the user: The generated document is sent to the terminal and displayed to the user. The user can review the displayed document and issue additional modification instructions as needed.

[1351] Specific example

[1352] For example, when a factory worker sets up assembly procedures for a new module, they upload a PDF file containing the basic instructions. The server analyzes the PDF file, extracts each step, and generates flow data. The worker then enters an instruction into the interactive interface, such as "Add a new module inspection." The server analyzes this instruction and updates the flow data. The updated flow chart and work manual are regenerated and displayed to the worker. The worker reviews it and provides final approval or additional instructions as needed.

[1353] Example of a prompt

[1354] "We have uploaded the initial assembly instructions."

[1355] "Add a new step: Module inspection"

[1356] "Please remove the inspection step from the assembly procedure."

[1357] Thus, the present invention makes it easy to automatically generate and update work flows and operating procedures within a factory, significantly reducing the workload for users.

[1358] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1359] Step 1:

[1360] The user uploads draft documents. These documents can be in formats such as PDF, Word, or Excel. The terminal provides an interface for selecting and submitting these draft documents, and sends the selected files to the server. The input is the file uploaded by the user, and the output is the draft document sent to the server.

[1361] Step 2:

[1362] The server analyzes the received draft data. Natural language processing techniques are used for the analysis to understand the context and structure of the data. Specifically, the "dbmdz / bert-large-cased-finetuned-conll03-english" model from the "transformers" library is used to extract each step of the business flow from the draft data. The input is the received draft data, and the output is the generated flow data.

[1363] Step 3:

[1364] The server initially generates a business flow based on the generated flow data. Here, each step and its sequence are defined and visualized as a graphical flowchart. This flowchart is displayed to the user in subsequent steps. The input is the flow data, and the output is the visualized business flow diagram.

[1365] Step 4:

[1366] The user inputs instructions into the system through an interactive interface. These instructions are entered in natural language and include adding new steps, modifying or deleting existing ones, and more. The input consists of the user's instructions (prompts), and the output is the content of the received instructions.

[1367] Step 5:

[1368] The server analyzes the user's instructions and updates the flow data accordingly. For example, if the instruction is "Add a new step: module inspection," the flow data will be updated to reflect its content and location. The input is the user's instructions and the current flow data, and the output is the updated flow data.

[1369] Step 6:

[1370] The server generates new business process diagrams and operation manuals based on the updated flow data. New diagrams and procedures containing specific details are generated, and this information is visualized. The input is the updated flow data, and the output is the new business process diagrams and operation manuals.

[1371] Step 7:

[1372] The terminal displays the generated business process diagrams and operation manuals to the user. The user can review the displayed materials and issue instructions for further changes as needed. The input is the new business process diagrams and operation manuals, and the output is the visualized materials displayed to the user.

[1373] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1374] This invention relates to a system that automatically generates and updates business flows and manuals based on initial drafts created by the user, and further incorporates an emotion engine that recognizes the user's emotions to provide responses adapted to the user.

[1375] System Configuration

[1376] This system consists of the following main components:

[1377] 1. How to upload draft documents

[1378] 2. Means for analyzing draft data and generating flow data

[1379] 3. Means of receiving instructions from the user

[1380] 4. Means for updating flow data based on instructions

[1381] 5. Means for generating documents that reflect updated flow data

[1382] 6. Means of displaying materials to users

[1383] 7. Emotion engine that recognizes user emotions

[1384] 8. Means for adjusting the system's response based on recognized emotions.

[1385] The program's specific operation

[1386] Upload of initial draft documents

[1387] The user selects and uploads their initial draft document (e.g., PDF, Word, or Excel file) through the system interface. The terminal then sends this file to the server. The sent file is temporarily stored on the server.

[1388] Data analysis and flow data generation

[1389] The server analyzes the received draft data. This analysis uses natural language processing techniques to understand the context and structure of the data (e.g., headings, paragraphs, lists). Based on the analysis results, flow data, which is a data model of the business process, is generated.

[1390] User change instructions

[1391] Users give instructions to the system through an interactive interface. This includes adding new steps, modifying or deleting existing steps, and more. Instructions are entered in natural language, so no special technical knowledge is required.

[1392] Use of the emotion engine

[1393] The emotion engine analyzes the user's voice and facial expressions to recognize their emotions. For example, it collects the user's voice tone and facial expressions through the camera and microphone when they input instructions, and analyzes their emotions. This helps determine whether the user is stressed or satisfied.

[1394] Flow data updates and response adjustments

[1395] The server integrates user instructions with the results of the emotion engine's analysis and updates the flow data accordingly. For example, if the user is stressed, the system displays more detailed guidance or additional support messages. Conversely, if the user is satisfied, it provides a concise response.

[1396] Generation and display of deliverables

[1397] Based on the updated flow data, new flow charts and manuals are generated. This process utilizes graphical drawing libraries and other tools. The generated materials and system responses are sent to the terminal and displayed to the user. The user can review the displayed materials and, if necessary, provide instructions for additional changes.

[1398] Specific example

[1399] For example, if a user wants to create a project management flow for the first time, they upload a document (e.g., a PDF) containing basic project steps as an initial draft. The server analyzes this document, extracts each step, and generates flow data.

[1400] Next, the user enters instructions into the dialogue interface, such as "Add a task to Phase 2." The server analyzes these instructions and updates the flow data accordingly. The emotion engine then analyzes the user's voice and facial expressions, and if it determines that the user is satisfied, it displays the updated flow diagram along with a relatively simple response. Conversely, if the user is stressed, the system provides more detailed explanations and guidance.

[1401] Thus, the present invention makes it possible to automatically generate and update business flows and manuals based on draft data, and furthermore, by recognizing user emotions and adjusting responses, it is possible to provide a more user-friendly system.

[1402] The following describes the processing flow.

[1403] Understood. The specific steps of the process are outlined below.

[1404] Step 1:

[1405] Users select and upload their initial draft documents (e.g., PDF, Word, or Excel files) through the system interface.

[1406] Step 2:

[1407] The terminal sends the draft data selected by the user to the server. The sent files are temporarily stored on the server.

[1408] Step 3:

[1409] The server analyzes the received data. Natural language processing techniques are used for the analysis to understand the context and structure of the data (e.g., headings, paragraphs, lists). Based on the analysis results, a data model (flow data) of the business process is generated.

[1410] Step 4:

[1411] The terminal displays the generated flow data to the user as a graphical flowchart. The user then reviews it.

[1412] Step 5:

[1413] Users give instructions to the system through an interactive interface, including adding new steps, modifying or deleting existing ones.

[1414] Step 6:

[1415] The emotion engine collects and analyzes the user's voice and facial expressions through input devices (microphone and camera). The emotion engine recognizes the user's emotional state (e.g., feeling stressed, feeling satisfied).

[1416] Step 7:

[1417] The device sends user instructions and emotion recognition results from the emotion engine to the server.

[1418] Step 8:

[1419] The server analyzes user instructions using natural language processing techniques and updates flow data accordingly. It also adjusts the system's response based on the results of the emotion engine (e.g., providing detailed guidance if the user is stressed).

[1420] Step 9:

[1421] The server generates new flowcharts and manuals based on the updated flow data. A graphical drawing library is used for this purpose.

[1422] Step 10:

[1423] Along with the newly generated flowcharts and manuals, responses tailored to the user's emotions are sent to the terminal. For example, a concise response if the user is satisfied, and a detailed guide if they are stressed.

[1424] Step 11:

[1425] The terminal displays the generated new flowcharts, manuals, and system responses to the user. The user can review the content and, if necessary, provide instructions for additional changes.

[1426] Step 12:

[1427] If the user provides additional instructions, steps 5 through 11 are repeated. Finally, once the user approves, the updated flowchart and manual are finalized.

[1428] This series of steps allows users to efficiently create and update workflows and manuals, and furthermore, the system's responses are adjusted based on sentiment recognition, resulting in a better user experience.

[1429] (Example 2)

[1430] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1431] Traditional automated systems for generating business flows and manuals have limited potential for improving the user experience because they provide uniform processing responses without considering user emotions or stress levels. Furthermore, a lack of convenience was a problem, as users often needed technical knowledge to issue instructions. Additionally, the difficulty for users to intuitively understand the update status and results of flow data was also an issue.

[1432] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1433] In this invention, the server includes means for uploading draft data, means for analyzing the draft data to generate flow data, and means for receiving instructions from the user. This makes it possible to recognize the user's emotions and stress levels and adjust the system response accordingly. Specifically, by including the analysis of the user's voice and facial expressions and adjusting the response based on the recognized emotions, the flow data update process becomes more user-friendly and the user can easily operate it with an intuitive interface.

[1434] A "draft document" is a document file format used by a system as the basis for analysis and data generation.

[1435] "Flow data" refers to a data model of a business process flow generated based on information analyzed from preliminary documents.

[1436] "User instructions" refer to input given by the user to the system, including the addition of new steps and the modification or deletion of existing steps.

[1437] "Means of recognizing emotions" refers to the function of a system that analyzes the user's voice and facial expressions to identify the user's emotional state.

[1438] "Means for adjusting responses" refers to a function that dynamically applies response content, such as confirmation messages and guidelines from the system, based on the recognized emotions of the user.

[1439] "Means of displaying materials to the user" refers to an interface for clearly displaying generated and updated flow data and related materials on the user's terminal.

[1440] This invention relates to a system that automatically generates and updates business flows and manuals based on initial drafts created by the user, and further incorporates an emotion engine that recognizes the user's emotions to provide responses adapted to the user.

[1441] System Configuration

[1442] This system consists of the following main components:

[1443] 1. How to upload draft documents

[1444] 2. Means for analyzing draft data and generating flow data

[1445] 3. Means of receiving instructions from the user

[1446] 4. Means for updating flow data based on instructions

[1447] 5. A means of recognizing emotions by analyzing the user's voice and facial expressions.

[1448] 6. Means for adjusting the system's response based on recognized emotions.

[1449] 7. Means for generating documents that reflect updated flow data

[1450] 8. Means for displaying materials to users

[1451] Detailed explanation of the program's processing

[1452] Upload of initial draft documents

[1453] The user selects and uploads a draft document (e.g., PDF, Word, or Excel file) created initially through the terminal's interface. The terminal sends this file to the server. The server stores the uploaded file in a temporary storage area.

[1454] Data analysis and flow data generation

[1455] The server analyzes the received draft data. This analysis uses natural language processing techniques (e.g., SpaCy, NLTK) to understand the context and structure of the data (headings, paragraphs, lists, etc.). Based on the analysis results, flow data, which is a data model of the business process, is generated.

[1456] User change instructions

[1457] The user provides instructions to the system through an interactive interface. This includes adding new steps, modifying or deleting existing ones. Instructions are entered in natural language, so no special technical knowledge is required. The terminal receives the user's input and sends it to the server.

[1458] Use of the emotion engine

[1459] The server receives user voice and facial expression data from the terminal and recognizes the user's emotions using an emotion engine. The emotion engine uses voice analysis (e.g., speech-to-text services) and facial expression analysis (e.g., OpenCV).

[1460] Flow data updates and response adjustments

[1461] The server integrates user instructions and sentiment analysis results, updating flow data accordingly. If the user is experiencing stress, the system displays detailed guidance and additional support messages. Conversely, if the user is satisfied, it provides a concise response.

[1462] Generation and display of deliverables

[1463] Based on the updated flow data, new flowcharts and manuals are generated. This process utilizes graphical drawing libraries (e.g., D3.js). The generated materials and system responses are sent to the terminal and displayed to the user.

[1464] Specific example

[1465] For example, if a user wants to create a project management flow, they would follow these steps:

[1466] The user uploads a PDF document outlining basic project steps as an initial draft. The server analyzes the document, extracts each step, and generates flow data. Next, the user inputs instructions, such as "Add a task to Phase 2," into the interactive interface. The server analyzes these instructions and updates the flow data accordingly. Subsequently, the emotion engine analyzes the user's voice and facial expressions, and if it determines that the user is satisfied, it displays an updated flow chart with a relatively simple response. Conversely, if the user is stressed, the system provides more detailed explanations and guidance.

[1467] Examples of prompts for generative AI models

[1468] "As an initial step in project management, please generate the following flow data. If you wish to add tasks to each step of Phase 1 and Phase 2, please provide appropriate instructions. Also, please adjust support messages according to the user's sentiment."

[1469] The above describes embodiments for carrying out the present invention.

[1470] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1471] Step 1:

[1472] Input: Initial draft documents created by the user (PDF, Word, Excel files)

[1473] Specific operation: The user clicks the "Upload" button on the device interface. The device sends the selected file to the server as an HTTP POST request.

[1474] Output: The uploaded file is saved to the server.

[1475] Step 2:

[1476] Input: Uploaded draft document

[1477] Specific operation: The server reads the received file and converts it into string data. Natural language processing techniques (e.g., SpaCy or NLTK) are used to analyze the document's context and structure (headings, paragraphs, lists, etc.).

[1478] Output: Flow data (JSON format) generated based on the analysis results.

[1479] Step 3:

[1480] Input: Flow data analysis results

[1481] Specific operation: The server displays initial flow data in the user interface. The user inputs instructions such as adding, modifying, or deleting steps in natural language through an interactive interface.

[1482] Output: User instructions are sent to the server via the terminal.

[1483] Step 4:

[1484] Input: User instructions

[1485] Specific operation: The server receives user instructions and analyzes the content of the instructions using natural language understanding (NLU) technology (e.g., the BERT model). Based on the analyzed instructions, the server updates the flow data.

[1486] Output: Updated flow data

[1487] Step 5:

[1488] Input: User voice and facial expression data

[1489] Specific operation: The device captures the user's voice and facial expressions through the microphone and camera and sends them to the server. The server converts the voice data into text using a speech-to-text service and recognizes emotions using facial expression analysis technology (e.g., OpenCV).

[1490] Output: Emotion recognition result

[1491] Step 6:

[1492] Input: Emotion recognition result

[1493] Specific operation: The server adjusts its response based on the emotion recognition results and user instructions. For example, if the user is stressed, it generates detailed guides or additional support messages; if the user is satisfied, it provides a concise response.

[1494] Output: Adjusted response

[1495] Step 7:

[1496] Input: Updated flow data

[1497] Specific operation: The server uses a graphical drawing library (e.g., D3.js) to generate new flowcharts and manuals based on the updated flow data.

[1498] Output: Generated flowcharts and manuals

[1499] Step 8:

[1500] Input: Generated flowcharts and manuals, and adjusted responses.

[1501] Specific operation: The server sends these deliverables to the terminal. The terminal displays the received materials and responses on the screen.

[1502] Output: Flowcharts, manuals, and system response messages displayed in a user-readable format.

[1503] The above describes the processing steps of this system.

[1504] (Application Example 2)

[1505] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1506] The creation and updating of conventional work flows and manuals is extremely time-consuming and difficult to address in a way that takes into account the emotions and stress levels of workers. As a result, work efficiency may decrease, and worker satisfaction may also decline. This problem is particularly serious in workplaces such as factories, where real-time flow updates and immediate feedback are required. This invention aims to solve these problems.

[1507] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for uploading draft data, means for analyzing the draft data to generate business flow data, means for receiving instructions from the user, means for updating the business flow data based on the instructions, means for recognizing the user's emotions using emotion recognition means, means for adjusting the response based on the recognized emotions, means for generating a document that reflects the updated business flow data, and means for displaying the document to the user. This enables the automatic and efficient generation and updating of business flows and manuals, as well as flexible responses that respond to the emotions of the workers.

[1508] A "draft document" is a document containing the basic content that was created initially.

[1509] "Business process flow data" refers to a data model that represents the flow and procedures of a business process.

[1510] "User instructions" refer to commands given by the user to the system, such as additions, deletions, and modifications.

[1511] "Emotion recognition means" refers to a function that analyzes the user's voice and facial expressions to recognize their emotions.

[1512] "Means of adjusting responses" refers to a function that optimizes the system's response based on recognized emotions.

[1513] "Means for generating documents" refers to a function that automatically creates new documents based on analyzed and updated flow data.

[1514] "Means of displaying materials to users" refers to a function that displays generated materials in a format that users can review.

[1515] This invention relates to a system that automatically generates and updates business flow data based on a draft document created by the user initially. The system incorporates a function that recognizes the user's emotions and adjusts the response accordingly.

[1516] System Configuration

[1517] This system consists of the following main components:

[1518] 1. How to upload draft documents

[1519] 2. A means of generating business flow data by analyzing draft documents.

[1520] 3. Means of receiving instructions from the user

[1521] 4. Means for recognizing a user's emotions using emotion recognition means

[1522] 5. Means of adjusting responses based on perceived emotions

[1523] 6. Means for generating documents that reflect updated business flow data

[1524] 7. Means for displaying the aforementioned materials to the user

[1525] Specific hardware and software to be used

[1526] hardware

[1527] Factory robots: Responsible for generating and displaying work instructions and documents.

[1528] Camera: Used to capture the user's facial expressions and recognize emotions.

[1529] Microphone: Used to capture the user's voice and recognize emotions and instructions.

[1530] Tablets and monitors: Used to visually display updated workflow data.

[1531] software

[1532] Natural language processing libraries (e.g., SpaCy): Used to analyze uploaded draft documents and generate business flow data.

[1533] Emotion recognition engine (e.g., Microsoft Azure Face API, Google Cloud Speech-to-Text): Used to analyze and recognize the user's emotions.

[1534] Flow data generation and update libraries (e.g., Graphviz): Used to generate data models of business processes and to draw visual flow diagrams based on updated data.

[1535] Flow of operations

[1536] The user uploads a draft document (e.g., PDF, Word, Excel file) they initially created. This document is sent to the server by the system and temporarily stored. The server uses a natural language processing library to analyze the document and generate business flow data. Next, the user provides instructions through an interactive interface, which are entered in natural language. These instructions are then analyzed, and the business flow data is updated.

[1537] Furthermore, emotion recognition measures analyze the user's voice and facial expressions to recognize their emotions. Based on these results, the response is adjusted. For example, if the user is feeling stressed, a detailed support message is provided. On the other hand, if the user is satisfied, a concise response is provided.

[1538] Based on updated workflow data, new documents are generated and displayed on tablets and monitors by factory robots.

[1539] Examples of specific cases and prompt statements

[1540] For example, consider a scenario where a factory worker uploads a work procedure manual they created for the first time. After uploading a PDF as a draft document to the system, the system analyzes it and generates business flow data. When the worker gives instructions in natural language, such as "Please add a new task to the next step," these instructions are analyzed and the business flow data is updated accordingly.

[1541] The emotion recognition system analyzes the worker's voice and facial expressions, and if the worker is experiencing stress, the system displays detailed guidance or additional support messages. Conversely, if the worker is satisfied, it provides a simple response.

[1542] The updated flowchart is displayed on a tablet or monitor, allowing workers to review it and provide additional instructions as needed.

[1543] Example of a prompt

[1544] "Please upload the work procedure manual."

[1545] "Add a new task to the next step."

[1546] "We are analyzing the emotions of the workers through cameras and microphones."

[1547] "We are adjusting our response based on the analysis results."

[1548] Thus, the present invention enables the automatic and efficient generation and updating of business flows and manuals, and also allows for flexible responses that respond to the emotions of the workers.

[1549] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1550] Step 1:

[1551] The user uploads draft documents (e.g., PDF, Word, Excel files) via their device. These draft documents are sent to the server and temporarily stored. The input is the draft document in file format, and the output is the saved draft document file.

[1552] Step 2:

[1553] The server analyzes the uploaded draft data using a natural language processing library (e.g., SpaCy). The analysis generates business flow data. The input is the saved draft data file, and the output is the generated business flow data.

[1554] Specific operations include text extraction and contextual analysis of the draft materials.

[1555] Step 3:

[1556] The user inputs instructions in natural language through an interactive interface. These instructions include adding new steps, modifying existing steps, and deleting existing ones. The input is the user's instructions, and the output is the analysis result.

[1557] Step 4:

[1558] The server analyzes user instructions and updates the business flow data. Natural language processing is also used for this analysis. The input is the analyzed user instructions, and the output is the updated business flow data.

[1559] Specific actions include extracting instructions and modifying corresponding business flow data.

[1560] Step 5:

[1561] The server analyzes the user's voice and facial expressions using emotion recognition tools to recognize their emotions. This analysis uses emotion recognition engines (e.g., Microsoft Azure Face API, Google Cloud Speech-to-Text). The input is the user's voice and facial expression data, and the output is the recognized emotion.

[1562] Step 6:

[1563] The server adjusts its response based on the perceived emotion. Specifically, it provides a detailed support message if the user is stressed, and a concise response if they are satisfied. The input is the perceived emotion, and the output is the adjusted response.

[1564] Step 7:

[1565] The server generates new documents based on updated business process flow data and displays them on tablets and monitors via factory robots. A flow data generation and updating library (e.g., Graphviz) is used for generation. The input is the updated business process flow data, and the output is the generated documents (flowcharts and manuals).

[1566] Step 8:

[1567] The user reviews the displayed materials and provides additional instructions as needed. The input is the generated materials, and the output is the user's new instructions.

[1568] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1569] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1570] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1571] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1572] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1573] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1574] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1575] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1576] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1577] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1578] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1579] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1580] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1581] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1582] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1583] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1584] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1585] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1586] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1587] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1588] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[1589] The following is further disclosed regarding the embodiments described above.

[1590] Understood. Below, I propose a draft of the claims regarding the distinctive features of the system.

[1591] (Claim 1)

[1592] A means of uploading draft documents,

[1593] A means for analyzing the aforementioned draft data and generating flow data,

[1594] Means of receiving instructions from users,

[1595] Means for updating the flow data based on the aforementioned instructions,

[1596] A means of generating documents that reflect updated flow data,

[1597] A means for displaying the aforementioned material to the user,

[1598] A system that includes this.

[1599] (Claim 2)

[1600] The system according to claim 1, characterized in that the aforementioned draft document is a PDF, Word, or Excel file.

[1601] (Claim 3)

[1602] The system according to claim 1, characterized in that the user's instructions are input in natural language.

[1603] "Example 1"

[1604] (Claim 1)

[1605] A means of uploading draft documents,

[1606] A means for analyzing the aforementioned draft data and generating flow data,

[1607] Means of receiving instructions from users,

[1608] Means for updating the flow data based on the aforementioned instructions,

[1609] A means of generating deliverables based on updated flow data,

[1610] A means of displaying the generated deliverables to the user,

[1611] A system that includes this.

[1612] (Claim 2)

[1613] The system according to claim 1, characterized in that the aforementioned draft material is any of the document files.

[1614] (Claim 3)

[1615] The system according to claim 1, characterized in that the user's instructions are input in natural language.

[1616] (Claim 4)

[1617] The system according to claim 1, characterized in that the analysis means uses natural language processing technology.

[1618] (Claim 5)

[1619] The system according to claim 1, characterized in that the output generation means uses a graphical display library.

[1620] "Application Example 1"

[1621] (Claim 1)

[1622] A means of uploading draft documents,

[1623] A means for analyzing the aforementioned draft data and generating flow data,

[1624] Means of receiving instructions from users,

[1625] Means for updating the flow data based on the aforementioned instructions,

[1626] A means of generating documents that reflect updated flow data,

[1627] A means for displaying the aforementioned material to the user,

[1628] This is an application installed on industrial machinery that uses natural language processing technology to analyze draft data uploaded by field workers and automatically generates workflows and operating procedures.

[1629] Based on user instructions, this workflow and operating procedures are updated, and a means is provided to graphically display the operation of industrial machinery and the workflow.

[1630] A system that includes this.

[1631] (Claim 2)

[1632] The system according to claim 1, characterized in that the aforementioned draft document is a PDF, Word, or Excel file.

[1633] (Claim 3)

[1634] The system according to claim 1, characterized in that the user's instructions are input in natural language.

[1635] "Example 2 of combining an emotion engine"

[1636] (Claim 1)

[1637] A means of uploading draft documents,

[1638] A means for analyzing the aforementioned draft data and generating flow data,

[1639] Means of receiving instructions from users,

[1640] Means for updating the flow data based on the aforementioned instructions,

[1641] A method for recognizing emotions by analyzing the user's voice and facial expressions,

[1642] Means for adjusting the system's response based on the recognized emotion,

[1643] A means of generating documents that reflect updated flow data,

[1644] A means for displaying the aforementioned material to the user,

[1645] A system that includes this.

[1646] (Claim 2)

[1647] The system according to claim 1, characterized in that the aforementioned draft data is in document file format.

[1648] (Claim 3)

[1649] The system according to claim 1, characterized in that the user's instructions are input in natural language.

[1650] "Application example 2 when combining with an emotional engine"

[1651] (Claim 1)

[1652] A means of uploading draft documents,

[1653] A means for analyzing the aforementioned draft data to generate business flow data,

[1654] Means of receiving instructions from users,

[1655] A means for updating the business flow data based on the aforementioned instructions,

[1656] A means of recognizing a user's emotions using emotion recognition means,

[1657] Means of adjusting responses based on recognized emotions,

[1658] A means of generating documents that reflect updated business flow data,

[1659] A means for displaying the aforementioned material to the user,

[1660] A system that includes this.

[1661] (Claim 2)

[1662] The system according to claim 1, characterized in that the aforementioned draft material is a document file.

[1663] (Claim 3)

[1664] The system according to claim 1, characterized in that the user's instructions are input in natural language. [Explanation of Symbols]

[1665] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of uploading draft documents, A means for analyzing the aforementioned draft data and generating flow data, Means of receiving instructions from users, Means for updating the flow data based on the aforementioned instructions, A means of generating documents that reflect updated flow data, A means for displaying the aforementioned material to the user, A system that includes this.

2. The system according to claim 1, characterized in that the aforementioned draft document is a PDF, Word, or Excel file.

3. The system according to claim 1, characterized in that the user's instructions are input in natural language.

Images