system

Abstract

We provide the system. [Solution] It has natural language processing capabilities and means for analyzing the content of work documents, A means for selecting an appropriate machine control tool based on the analyzed work documents, Means for generating control codes for selected machine control tools, A means for applying the generated control code to the execution environment and reporting the results, Means for automating the functions of industrial facilities, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including: receiving a user utterance; adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot; encoding the prompt; and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] A series of operations from the execution of a manually time-consuming work instruction sheet to the selection, setting, verification, and subsequent documentation of an optimal automation tool require a great deal of human resources and time, hindering efficient operation. Furthermore, manual errors may occur in this process, and improvement in the accuracy and quality of work is required.

Means for Solving the Problems

[0005] This invention analyzes work instructions using natural language processing technology and selects the optimal automation tool based on their content. It then generates automation code for that tool and applies it to the execution environment to improve work efficiency and reduce errors. Furthermore, it identifies and presents tasks that are difficult to automate to the person in charge, and automatically generates updated work procedures based on the execution results, thereby providing a system that improves operational efficiency and quality.

[0006] "Natural language processing" refers to the technology of analyzing and understanding natural language text and extracting its intent and meaning.

[0007] A "work instruction sheet" is a document that describes the steps and procedures necessary to perform a specific process or operation.

[0008] "Means of analysis" refers to methods or devices for breaking down and analyzing information or data to extract important elements or patterns.

[0009] An "automation tool" is software or a system used to automatically perform tasks or operations that would normally be done manually.

[0010] "Means of selection" refers to a method or apparatus for determining the most suitable option from among multiple choices for a given purpose.

[0011] "Automation code" refers to programs or scripts written to allow a computer to automatically execute a set of predetermined procedures.

[0012] The "execution environment" refers to the software and hardware configuration necessary for a system or application to function.

[0013] "Means of application" refers to a method or apparatus for combining and applying solutions or technologies to a specific purpose.

[0014] "Execution result" refers to the effects or outputs that occur after a set operation or process has been executed.

[0015] "Responsible person" refers to a person who has the responsibility for a specific business or project and undertakes the role of execution.

[0016] "Work procedure document" is a document that details the specific procedures for business or work, enabling workers to accurately carry out their work by following it.

Brief Description of the Drawings

[0017] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of a data processing system in Example 1. [Figure 12]It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0018] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0019] First, the language used in the following description will be explained.

[0020] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0021] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0022] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0023] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0025] [First Embodiment]

[0026] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0027] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0030] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0033] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0037] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0038] This invention provides a system for performing the analysis and automation of work instructions, which were previously done manually. In implementing the invention, a server primarily plays a central role, analyzing work instructions using natural language processing technology and selecting the optimal automation tool to improve work efficiency.

[0039] Specifically, when a user uploads a work order to the server via their terminal, the server analyzes its contents using natural language processing technology. Based on the analysis results, the server selects the most suitable automation tool for the procedure and generates automation code corresponding to that tool. This code is then applied to the execution environment according to the system's instructions and used to automate the operations required in the work order.

[0040] As a concrete example of this system, consider a scenario where a user sends a "database backup procedure document" to the server. The server analyzes the document and selects an appropriate automation tool (such as a scripting tool or configuration management tool) to perform a safe and efficient backup operation on the database. Based on the selected tool, the server generates an appropriate script, automates the backup process, and reports the results to the user.

[0041] This simplifies complex and time-consuming tasks that were previously performed manually, leading to increased efficiency and improved quality. Furthermore, the server documents the results of each step and automatically generates work procedure documents, providing reference materials for subsequent tasks. This allows users to reduce management costs while enabling effective business operations.

[0042] The following describes the processing flow.

[0043] Step 1:

[0044] The user uploads the work order to the server using a terminal. The server receives the uploaded file and confirms its storage location.

[0045] Step 2:

[0046] The server activates its natural language processing function and analyzes the contents of the received work instructions. The analysis involves breaking down the procedures, extracting keywords, and identifying related work categories.

[0047] Step 3:

[0048] Based on the analysis results, the server selects the appropriate automation tool. For example, if scripting is required, it selects the appropriate scripting tool.

[0049] Step 4:

[0050] The server automatically generates code or scripts corresponding to the selected automation tool. This generated code includes content corresponding to each step of the work order.

[0051] Step 5:

[0052] The server applies the generated code to a specific execution environment. Before application, it runs the code in a test environment to check for errors, and if there are no problems, it is deployed to the production environment.

[0053] Step 6:

[0054] The server collects the execution results and provides them to the user as a report. It also automatically generates updated work procedures based on the results, preparing for future tasks.

[0055] Step 7:

[0056] The server identifies tasks that are difficult to automate and presents them to the user as a list. The user can then use this list to proceed with the tasks manually.

[0057] (Example 1)

[0058] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0059] In the process of analyzing and automating work instructions, there is a problem in quickly and accurately determining which procedures can be automated and which tools are best suited for that process. Furthermore, the process of identifying steps that require manual work and efficiently notifying the responsible personnel is cumbersome. In addition, after the execution of the generated automation scripts, properly recording the updated information reflecting the results and creating documentation to be used for subsequent tasks requires significant resources.

[0060] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0061] In this invention, the server includes means for receiving work instructions with natural language processing capabilities, means for analyzing the received work instructions to identify key procedures, and means for selecting the optimal automation software based on the analysis results. This enables improved analysis accuracy and a more efficient selection process. Furthermore, by detecting manual processes and quickly notifying the responsible personnel, overall work efficiency is improved. In addition, by generating immediately updated work instruction documents based on execution results, the effort required for document creation is reduced, and the overall quality of operations is improved.

[0062] "Natural language processing" is a technology that converts written text into a format that a computer can understand, and then analyzes and processes the information contained within it.

[0063] A "work instruction sheet" is a document that details how to perform a specific task or procedure.

[0064] "Automation software" is a program that converts tasks performed by humans into tasks that can be executed by machines, thereby automating business processes.

[0065] An "automation script" is a set of instructions or code created to automatically execute specific procedures or tasks.

[0066] "Operational infrastructure" refers to the system environment and infrastructure necessary for the normal execution and management of programs and scripts.

[0067] A "manual process" is a business process that requires direct human involvement because automation is difficult or impossible.

[0068] A "work instruction document" is a document that outlines the specific procedures and policies for carrying out a task, and is used as a reference for subsequent processes and related work.

[0069] This invention is a system for efficiently analyzing and automating work instructions. Its main components include a terminal, a server, a natural language processing model, and various automation software.

[0070] First, the user uploads the work order to the server using a terminal. The terminal consists of a typical computer or tablet device. In this process, the user selects files and sends them via a browser.

[0071] The server analyzes the received work instructions using a natural language processing model. Specific software used includes generative AI models such as BERT and GPT. The server tokenizes the text and extracts important work procedures and information by understanding the context.

[0072] Based on the analysis results, the server selects the most suitable automation software. For example, Ansible or Puppet, which are suitable for operational use, might be selected. After selection, the server generates an automation script tailored to the identified task. This includes formatting the script's syntax and structure for the chosen software.

[0073] The generated script is executed on the server's operational infrastructure. The execution results are recorded as logs and reported to the user. Furthermore, documentation is created based on the execution results, and a work instruction document that can be used as a reference for subsequent tasks is automatically generated.

[0074] For example, if a user sends a "database backup procedure document" to the server, the server analyzes its contents and selects and generates appropriate tools and scripts to ensure safe and efficient data backup. The generated scripts are executed on the server, and after successful completion, the results are reported to the user.

[0075] To use this system, you can input a prompt like the following into the generating AI model: "Analyze this text and suggest the best tools and scripts for automation." This allows the system to provide specific information to support decisions regarding automation.

[0076] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0077] Step 1:

[0078] The user uploads a work order to the server using a terminal. The input is the work order file selected by the user from the terminal. On the terminal, the user selects the target work order from a file selection dialog and clicks the send button. The output is the work order saved on the server. The server stores this file in a specified directory for analysis preparation.

[0079] Step 2:

[0080] The server analyzes the content of the uploaded work instructions using natural language processing technology. The input is the work instructions stored on the server in Step 1. The server uses a generative AI model to tokenize the text, understand the context, and perform data processing to extract important procedures and keywords. The output is structured data of the analyzed work instructions. This data is used in subsequent processing.

[0081] Step 3:

[0082] The server selects the optimal automation software based on the analysis results. The input is the analysis result data from step 2. The server performs a data calculation to select the most suitable automation tool from a list of automation tools according to the specified conditions. The output is the name of the selected automation software. This selection result then leads to script generation.

[0083] Step 4:

[0084] The server generates an automation script based on the selected automation software. The inputs are the names of the automation tools selected in step 3 and the analysis results from step 2. The server performs data processing to generate the script by combining the appropriate script syntax and structure. The output is the generated script file. This script then proceeds to the next execution process.

[0085] Step 5:

[0086] The server executes the generated script on the production platform. The input is the script generated in step 4. During execution, the server records operation logs and checks the success or failure of each step. The output is the execution result log data. This log is used for reporting work results and generating documentation.

[0087] Step 6:

[0088] The server automatically generates documents based on the execution results and reports them to the user. The input is the execution result log obtained in step 5. The server analyzes the execution results and processes the data to format it into a document. The output is the generated report and updated work instruction document. These are provided to the user and contribute to improving work efficiency.

[0089] (Application Example 1)

[0090] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0091] In modern industrial facilities, complex and diverse work processes exist, making efficient management and automation extremely difficult. In particular, the process of extracting appropriate procedures from work documents and implementing automation relies on manual work, which is time-consuming and labor-intensive. Furthermore, when procedures that are difficult to automate must be manually verified, the overall efficiency suffers. There is a need to solve these problems and provide a system that effectively automates work processes in industrial facilities while also enabling rapid responses to procedures that are difficult to automate.

[0092] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0093] In this invention, the server includes means for analyzing the content of work documents, means for selecting appropriate machine control tools based on the analyzed work documents, and means for generating control codes for the selected machine control tools. This enables the automation of industrial facility functions and the effective management of work processes. Furthermore, by extracting procedures that are difficult to automate based on the analysis results and presenting them to the person in charge, it enables the rapid and efficient supplementation of necessary manual work.

[0094] "Natural language processing" refers to the technology that allows computers to understand and process human language, enabling document analysis and information extraction.

[0095] A "work document" is a document that contains the procedures and instructions necessary to perform a specific task, and serves as a guide for efficient work in industrial facilities.

[0096] "Analysis" is the act of extracting structure and meaning from specific data or information and performing processing to gain a deeper understanding.

[0097] A "machine control tool" is a software tool used to efficiently and automatically operate and manage equipment and machinery within industrial facilities.

[0098] A "control code" is a set of instructions and programs necessary to operate a specific machine or device, and is a crucial element in realizing machine control.

[0099] The "execution environment" refers to the combination of hardware and software required for the generated control code to actually run.

[0100] A description of the embodiment for carrying out the invention will be provided.

[0101] This invention provides a system for efficiently automating work processes in industrial facilities. The user sends work documents to a server via a terminal. The server analyzes the documents using a program with natural language processing technology and selects appropriate machine control tools. Specifically, the server analyzes the documents using a natural language processing toolkit (e.g., spaCy or NLTK) and, based on the analysis results, selects the optimal tool for automation (e.g., Ansible or Docker). Based on the selected tool, the server generates control code. This control code is applied to the execution environment (e.g., a specified PLC or SCADA system) to control machinery or robots.

[0102] The control codes generated in this way are applied in the specified environment to automate the work process. The server reports the execution results to the user, allowing them to monitor the progress of the work. Furthermore, by identifying procedures that are difficult to automate based on the analysis results and informing the user, it becomes possible to efficiently supplement necessary manual work.

[0103] As a concrete example, this automated system manages the process of preparing tools and positioning parts according to a schedule during the regular maintenance of robot arms used on an assembly line. Users can use prompt statements such as the following:

[0104] Example of a prompt:

[0105] "Automate the process of checking the oil level of the robotic arm and refilling it as needed."

[0106] "I would like you to analyze the setup instructions for the tools needed for the next product change and automate the preparation process."

[0107] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0108] Step 1:

[0109] The server receives a work document from the user. This received document is the data to be analyzed for natural language processing. The server converts this document into internal data and prepares it for use in the next analysis step.

[0110] Step 2:

[0111] The server analyzes the received work documents using a natural language processing toolkit (e.g., spaCy or NLTK). The data processing performed here involves extracting keywords and structural information from the documents. As a result of the analysis, the work procedures and important elements are extracted and organized into specific instructions.

[0112] Step 3:

[0113] The server selects the optimal machine control tool based on the analysis results. Specifically, it searches for libraries of corresponding automation tools (e.g., Ansible or Docker) based on the extracted keywords and determines the most suitable tool. The input is the analysis results, and the output is the selected control tool.

[0114] Step 4:

[0115] The server generates control codes using the selected machine control tool. This generation process executes templates and scripts for the selected tool to generate specific control codes. The input is the selected tool, and the output is the generated control code.

[0116] Step 5:

[0117] The server applies the generated control code to the target execution environment. Specifically, it sends the code to the PLC or SCADA system of the industrial facility and performs the processing necessary to start the automated operation of the equipment. The output is the result of the operation in the execution environment.

[0118] Step 6:

[0119] The server analyzes the execution results and reports them to the user. Specifically, it determines whether the operation was completed successfully or if there were any errors based on the data obtained from the execution environment. The input is the execution result data, and the output is the report content.

[0120] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0121] This invention provides a system that combines natural language processing capabilities with an emotion engine to enable analysis of work instructions, selection and execution of automated processes, and flexible responses based on the user's emotional reactions. This system operates server-centric, aiming to improve user operational efficiency and reduce mental burden.

[0122] The user sends a work order to the server via their terminal. The server uses natural language processing to analyze the content of the work order and identify the scope of automation. Based on the analysis results, it then selects the optimal automation tool and generates the necessary automation code. The generated code is applied in the configured execution environment, and the work is performed automatically.

[0123] In addition, the server includes an emotion engine that measures the user's emotional state while working and uses that information to further optimize the process. Specifically, it provides a more comfortable working environment by suggesting ways to simplify processes or increasing support when the user is feeling stressed. For example, when automating the email server setup process, the system enhances support by providing additional explanations and generating more detailed instructions if the user expresses anxiety.

[0124] In this way, the present invention can integrate analyzed data with emotional feedback to improve work efficiency and user experience. As a result, it achieves improved automation accuracy and quality of work procedures, reducing user errors and burdens.

[0125] The following describes the processing flow.

[0126] Step 1:

[0127] The user uploads the work order to the server using their terminal. The server starts operating in response to this action, receiving and saving the work order file.

[0128] Step 2:

[0129] The server activates its natural language processing function to analyze the contents of the work instructions. The analysis extracts important keywords and operational steps, and categorizes the procedures based on these.

[0130] Step 3:

[0131] Based on the analysis results, the server selects the most suitable automation tool. Depending on the nature and scope of the procedure, it selects tools such as those for system configuration changes or test automation.

[0132] Step 4:

[0133] The server generates automation code corresponding to the selected automation tool. The generated code includes specific operational instructions based on the work order and is formatted into an executable form.

[0134] Step 5:

[0135] The server applies the automated code generated in the execution environment. Before application, it verifies that it works correctly in the test environment, and if there are no problems, it is deployed to the production environment.

[0136] Step 6:

[0137] The server uses an emotion engine to monitor the user's emotional state while they are working. While the user interacts with the interface, it analyzes camera footage and input patterns to infer their feelings.

[0138] Step 7:

[0139] Based on the analysis results of the emotion engine, the server provides support tailored to the user. For example, if the user is feeling frustrated, it may present a tutorial or simplify an automated process.

[0140] Step 8:

[0141] After all processes are complete, the server generates a report of the execution results and provides it to the user. Furthermore, it updates the work procedure manual based on user feedback to improve future operations.

[0142] (Example 2)

[0143] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0144] Conventional automation technologies were limited to analyzing work instructions and generating and executing automation code, and were unable to respond flexibly while considering the user's emotional state. This could lead to user stress, making it difficult to improve work efficiency and usability. Furthermore, decisions regarding parts that were difficult to automate and updates to procedures based on execution results were often performed manually, resulting in a significant burden.

[0145] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0146] In this invention, the server includes means for analyzing the contents of an information instruction document, means for selecting an appropriate automation device based on the analyzed information instruction document, means for generating an automation program for the selected automation device, and means for evaluating the user's emotional state and making suggestions to optimize the work process based on that evaluation. This enables flexible automation that responds to the user's emotions, thereby improving work efficiency and usability.

[0147] "Natural language processing" refers to the technology used by computers to understand and analyze human language and extract its meaning.

[0148] An "information instruction sheet" is a document that contains instructions and details necessary for a task or process, written in natural language.

[0149] "Automated equipment" is a general term for hardware or software that automatically performs specified tasks according to a program.

[0150] An "automation program" is software code or a script designed to perform a specific task without human intervention.

[0151] The "implementation environment" refers to the environment, including the computer resources and system settings necessary for the automation program to run.

[0152] "Assessing a user's emotional state" is the process of detecting a user's emotional response and quantifying or categorizing that state.

[0153] "Suggestions for optimizing the work process" refer to improvement measures or alternative procedures instructed by the system to enhance the user's work efficiency and comfort.

[0154] This invention combines natural language processing capabilities with an emotion evaluation engine to provide analysis of work instructions, selection and execution of automated processes, and adaptive support based on user emotions. It operates server-centric, and the overall process is carried out as follows:

[0155] The user first sends a work instruction sheet to the server via their terminal. Since this instruction sheet is written in natural language, it can be used by users without special technical knowledge. The server receives the instruction sheet and analyzes it using natural language processing capabilities. For this analysis, a general-purpose machine learning library, for example, is used as the natural language processing library.

[0156] The server selects the appropriate automation device based on the analysis results. During the selection process, commonly used automation software, such as RPA solutions, may be utilized. Subsequently, an automation program is generated for the selected tool. This program is automatically created in script format and applied in the configured execution environment.

[0157] Furthermore, the server is equipped with technology for evaluating user emotions. This includes techniques such as facial recognition and voice analysis, and in some cases, a general-purpose emotion analysis API. If the server determines that a user is experiencing stress or anxiety during their work, it provides additional instructions and support. For example, it might display detailed guides in the user interface or provide supportive video instructions.

[0158] As a concrete example, suppose a user sends a work order to the server stating, "Automatically create the folder structure for a new project." In this case, the server analyzes the order, selects the appropriate folder generation program, and executes it in the configured environment. If the user expresses any concerns along the way, the server displays detailed instructions on the screen to help the user understand.

[0159] An example of a prompt message might be: "Please automate the setup process for new mail servers. Also, please provide detailed instructions for users who may feel unsure."

[0160] This embodiment of the invention makes it possible to realize an automated work process that is easy for users to operate and reduces their mental burden.

[0161] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0162] Step 1:

[0163] The user inputs a work instruction document written in natural language using a terminal and sends it to the server. This input is a document clearly describing the specific work to be done. The user's actions include creating and sending the instruction document. The server receives this instruction document and proceeds to the next analysis step.

[0164] Step 2:

[0165] The server analyzes received work orders using natural language processing capabilities. The input is the work order received from the user, and the output is the analyzed specific task information. Specific data processing includes text segmentation, keyword extraction, and contextual understanding. Based on the analysis results, the server determines which parts can be automated.

[0166] Step 3:

[0167] The server selects the most suitable automation device based on the analysis results. It receives the analyzed task information as input and the selected automation device as output. In this step, it searches the database for corresponding automation tools and scripts and selects the tool that can perform the task most efficiently.

[0168] Step 4:

[0169] The server generates automation programs for the selected automation devices. It takes the selection results as input and generates specific program code as output. Data calculations include script assembly and automatic generation of API calls.

[0170] Step 5:

[0171] The generated automation program is executed in the implementation environment. The server applies the program code to the implementation environment and automatically performs the configured tasks. The output is the result of the automated task execution, and the results are recorded in a predetermined format.

[0172] Step 6:

[0173] The server evaluates the user's emotional state during operation. Input is data such as the user's facial expressions and voice, and output is the evaluation result of the emotional state. Specific operations include real-time data capture and analysis.

[0174] Step 7:

[0175] Based on the user's emotional assessment, the server proposes optimizations to the work process. The emotional state assessment is taken as input, and the output is the optimized proposal. For example, if the server determines that the work is complex and stressful, it will offer suggestions to simplify the work procedure.

[0176] (Application Example 2)

[0177] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0178] In modern manufacturing, while there is a demand for automating processes based on work instructions, the challenge lies in reducing the mental burden and stress on workers and enabling them to perform tasks efficiently and flexibly. In particular, when workers feel anxious about complex instructions, there is a lack of appropriate procedure optimization and emotionally responsive support, and this problem needs to be addressed.

[0179] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0180] In this invention, the server includes means for analyzing the contents of work instructions, means for selecting an appropriate automation tool based on the analyzed work instructions, means for generating automation code for the selected automation tool, and means for acquiring emotional states and optimizing work procedures based on that information. This enables the selection and execution of automation processes from the analysis of work instructions, and further enables the provision of appropriate support according to the emotional state of the worker, thereby realizing an efficient and less stressful work environment.

[0181] "Natural language processing" refers to technologies that analyze input text data and understand its meaning and intent.

[0182] A "work instruction sheet" is a document that describes the procedures and conditions necessary to perform a specific task.

[0183] An "automation tool" is software or a system used to efficiently perform a specific task.

[0184] "Automation code" is a program written to execute a process automatically.

[0185] The "execution environment" refers to the hardware and software configuration required for a program to run.

[0186] "Emotional state" refers to a user's psychological or emotional condition, and measuring its changes is an indicator that can be used to support their work.

[0187] "Optimization" refers to improving existing processes and procedures to achieve a specific objective and maximize efficiency.

[0188] A system for implementing this invention consists of a server, a user terminal, and a work environment.

[0189] Program Overview

[0190] The server executes specific libraries (e.g., spaCy and Transformers) to perform natural language processing and analyzes work instructions received from the user's terminal. Based on the analysis results, it selects appropriate automation tools and generates program code. This automation code can then run in an execution environment built on the server. Furthermore, the server uses an emotion analysis engine (e.g., IBM Watson®) to evaluate the user's emotional state in real time and optimize the work procedure accordingly. For example, if the user is feeling anxious, it provides additional explanations or support.

[0191] Hardware and software used

[0192] Server: This is the central unit that analyzes work instructions and selects automation tools.

[0193] User terminal (e.g., smart glasses): A device used by workers to send work instructions to the server.

[0194] Emotion analysis engine: This is software used to measure a user's emotional state.

[0195] Specific example

[0196] Consider a scenario on a manufacturing line where a user puts on smart glasses and begins the setup procedure for a new machine. The user takes a picture of the work instructions through the glasses and sends them to a server. The server analyzes the information, generates the necessary automation code, and executes it. If the emotion engine detects the user's anxiety, it provides more specific setup instructions or visual guides to the glasses to aid understanding.

[0197] Example of a prompt

[0198] "Please explain the assembly procedure for the new parts. If there are any parts of the procedure that are complex or difficult to understand, please provide a detailed explanation."

[0199] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0200] Step 1:

[0201] The user uses smart glasses to photograph the work instruction sheet. At this time, the user captures the image of the work instruction sheet into their device and sends it to the server as digital data. The input is the image of the work instruction sheet, which forms the basis for subsequent data processing.

[0202] Step 2:

[0203] The server applies OCR technology to the received image of the work instruction and converts it into text data. This is data processing to convert image data into string data. The output is the text data of the work instruction.

[0204] Step 3:

[0205] The server uses a natural language processing library to analyze the text data of the work instructions in detail. This involves semantic and syntactic analysis of the text to identify tasks that can be automated. The input to this process is text data obtained by OCR, and the output is information about the analyzed tasks.

[0206] Step 4:

[0207] Based on the analyzed information, the server selects the optimal automation tool and generates the necessary automation code. The input is the task information obtained in step 3, and the output code is associated with a specific tool. At this stage, the server performs data calculations and creates code aimed at efficient task execution.

[0208] Step 5:

[0209] The generated automation code is run in a specific execution environment on the server. Here, the code is applied and its results are retrieved. The output of this process is the execution results and their report. The server monitors the execution process and collects results as needed.

[0210] Step 6:

[0211] The server uses an emotion analysis engine to monitor and evaluate the user's emotional state in real time. It analyzes the user's stress level and provides simplified procedures or additional explanations as needed. Inputs include the user's biometric data and facial expressions, and the analysis results are the output.

[0212] Step 7:

[0213] The user reviews the optimized instructions displayed on the smart glasses and proceeds with the task. Based on the sentiment analysis results obtained in Step 6, the server provides improvement suggestions, and the user utilizes specific work support. At this stage, the outputted instructions are presented in a way that is appropriate to the work environment.

[0214] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0215] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0216] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0217] [Second Embodiment]

[0218] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0219] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0220] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0221] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0222] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0223] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0224] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0225] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0226] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0227] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0228] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0229] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0230] This invention provides a system for performing the analysis and automation of work instructions, which were previously done manually. In implementing the invention, a server primarily plays a central role, analyzing work instructions using natural language processing technology and selecting the optimal automation tool to improve work efficiency.

[0231] Specifically, when a user uploads a work order to the server via their terminal, the server analyzes its contents using natural language processing technology. Based on the analysis results, the server selects the most suitable automation tool for the procedure and generates automation code corresponding to that tool. This code is then applied to the execution environment according to the system's instructions and used to automate the operations required in the work order.

[0232] As a concrete example of this system, consider a scenario where a user sends a "database backup procedure document" to the server. The server analyzes the document and selects an appropriate automation tool (such as a scripting tool or configuration management tool) to perform a safe and efficient backup operation on the database. Based on the selected tool, the server generates an appropriate script, automates the backup process, and reports the results to the user.

[0233] This simplifies complex and time-consuming tasks that were previously performed manually, leading to increased efficiency and improved quality. Furthermore, the server documents the results of each step and automatically generates work procedure documents, providing reference materials for subsequent tasks. This allows users to reduce management costs while enabling effective business operations.

[0234] The following describes the processing flow.

[0235] Step 1:

[0236] The user uploads the work order to the server using a terminal. The server receives the uploaded file and confirms its storage location.

[0237] Step 2:

[0238] The server activates its natural language processing function and analyzes the contents of the received work instructions. The analysis involves breaking down the procedures, extracting keywords, and identifying related work categories.

[0239] Step 3:

[0240] Based on the analysis results, the server selects the appropriate automation tool. For example, if scripting is required, it selects the appropriate scripting tool.

[0241] Step 4:

[0242] The server automatically generates code or scripts corresponding to the selected automation tool. This generated code includes content corresponding to each step of the work order.

[0243] Step 5:

[0244] The server applies the generated code to a specific execution environment. Before application, it runs the code in a test environment to check for errors, and if there are no problems, it is deployed to the production environment.

[0245] Step 6:

[0246] The server collects the execution results and provides them to the user as a report. It also automatically generates updated work procedures based on the results, preparing for future tasks.

[0247] Step 7:

[0248] The server identifies tasks that are difficult to automate and presents them to the user as a list. The user can then use this list to proceed with the tasks manually.

[0249] (Example 1)

[0250] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0251] In the process of analyzing and automating work instructions, there is a problem in quickly and accurately determining which procedures can be automated and which tools are best suited for that process. Furthermore, the process of identifying steps that require manual work and efficiently notifying the responsible personnel is cumbersome. In addition, after the execution of the generated automation scripts, properly recording the updated information reflecting the results and creating documentation to be used for subsequent tasks requires significant resources.

[0252] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0253] In this invention, the server includes means for receiving work instructions with natural language processing capabilities, means for analyzing the received work instructions to identify key procedures, and means for selecting the optimal automation software based on the analysis results. This enables improved analysis accuracy and a more efficient selection process. Furthermore, by detecting manual processes and quickly notifying the responsible personnel, overall work efficiency is improved. In addition, by generating immediately updated work instruction documents based on execution results, the effort required for document creation is reduced, and the overall quality of operations is improved.

[0254] "Natural language processing" is a technology that converts written text into a format that a computer can understand, and then analyzes and processes the information contained within it.

[0255] A "work instruction sheet" is a document that details how to perform a specific task or procedure.

[0256] "Automation software" is a program that converts tasks performed by humans into tasks that can be executed by machines, thereby automating business processes.

[0257] An "automation script" is a set of instructions or code created to automatically execute specific procedures or tasks.

[0258] "Operational infrastructure" refers to the system environment and infrastructure necessary for the normal execution and management of programs and scripts.

[0259] A "manual process" is a business process that requires direct human involvement because automation is difficult or impossible.

[0260] A "work instruction document" is a document that outlines the specific procedures and policies for carrying out a task, and is used as a reference for subsequent processes and related work.

[0261] This invention is a system for efficiently analyzing and automating work instructions. Its main components include a terminal, a server, a natural language processing model, and various automation software.

[0262] First, the user uploads the work order to the server using a terminal. The terminal consists of a typical computer or tablet device. In this process, the user selects files and sends them via a browser.

[0263] The server analyzes the received work instructions using a natural language processing model. Specific software used includes generative AI models such as BERT and GPT. The server tokenizes the text and extracts important work procedures and information by understanding the context.

[0264] Based on the analysis results, the server selects the most suitable automation software. For example, Ansible or Puppet, which are suitable for operational use, might be selected. After selection, the server generates an automation script tailored to the identified task. This includes formatting the script's syntax and structure for the chosen software.

[0265] The generated script is executed on the server's operational infrastructure. The execution results are recorded as logs and reported to the user. Furthermore, documentation is created based on the execution results, and a work instruction document that can be used as a reference for subsequent tasks is automatically generated.

[0266] For example, if a user sends a "database backup procedure document" to the server, the server analyzes its contents and selects and generates appropriate tools and scripts to ensure safe and efficient data backup. The generated scripts are executed on the server, and after successful completion, the results are reported to the user.

[0267] To use this system, you can input a prompt like the following into the generating AI model: "Analyze this text and suggest the best tools and scripts for automation." This allows the system to provide specific information to support decisions regarding automation.

[0268] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0269] Step 1:

[0270] The user uploads a work order to the server using a terminal. The input is the work order file selected by the user from the terminal. On the terminal, the user selects the target work order from a file selection dialog and clicks the send button. The output is the work order saved on the server. The server stores this file in a specified directory for analysis preparation.

[0271] Step 2:

[0272] The server analyzes the content of the uploaded work instructions using natural language processing technology. The input is the work instructions stored on the server in Step 1. The server uses a generative AI model to tokenize the text, understand the context, and perform data processing to extract important procedures and keywords. The output is structured data of the analyzed work instructions. This data is used in subsequent processing.

[0273] Step 3:

[0274] The server selects the optimal automation software based on the analysis results. The input is the analysis result data from step 2. The server performs a data calculation to select the most suitable automation tool from a list of automation tools according to the specified conditions. The output is the name of the selected automation software. This selection result then leads to script generation.

[0275] Step 4:

[0276] The server generates an automation script based on the selected automation software. The inputs are the names of the automation tools selected in step 3 and the analysis results from step 2. The server performs data processing to generate the script by combining the appropriate script syntax and structure. The output is the generated script file. This script then proceeds to the next execution process.

[0277] Step 5:

[0278] The server executes the generated script on the production platform. The input is the script generated in step 4. During execution, the server records operation logs and checks the success or failure of each step. The output is the execution result log data. This log is used for reporting work results and generating documentation.

[0279] Step 6:

[0280] The server automatically generates documents based on the execution results and reports them to the user. The input is the execution result log obtained in step 5. The server analyzes the execution results and processes the data to format it into a document. The output is the generated report and updated work instruction document. These are provided to the user and contribute to improving work efficiency.

[0281] (Application Example 1)

[0282] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0283] In modern industrial facilities, there are complex and diverse work processes, and it is extremely difficult to manage and automate them efficiently. In particular, the process of extracting appropriate procedures from work documents and executing automation depends on manual work, which requires a great deal of time and labor. In addition, when there are procedures that are difficult to automate, the handling methods must be manually checked, resulting in a problem of reduced overall efficiency. There is a need to solve these problems and provide a mechanism that can effectively automate work processes in industrial facilities and quickly respond to procedures that are difficult to automate.

[0284] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0285] In this invention, the server has a natural language processing function and includes means for analyzing the content of a work document, means for selecting an appropriate machine control tool based on the analyzed work document, and means for generating a control code for the selected machine control tool. Thereby, it becomes possible to automate the functions of an industrial facility and effectively manage work processes. In addition, by extracting procedures that are difficult to automate based on the analysis results and presenting them to the person in charge, it is realized to quickly and efficiently complement the necessary manual work.

[0286] The "natural language processing function" is a technology that enables a computer to understand and process human language and is a function that enables document analysis and information extraction.

[0287] A "work document" is a document in which the procedures and instructions necessary to perform a specific work are described and serves as a guideline for efficient work in an industrial facility.

[0288] "Analysis" is an act of extracting the structure and meaning from specific data or information and performing a process to obtain a deeper understanding.

[0289] A "machine control tool" is a software tool used to efficiently and automatically operate and manage equipment and machinery within industrial facilities.

[0290] A "control code" is a set of instructions and programs necessary to operate a specific machine or device, and is a crucial element in realizing machine control.

[0291] The "execution environment" refers to the combination of hardware and software required for the generated control code to actually run.

[0292] A description of the embodiment for carrying out the invention will be provided.

[0293] This invention provides a system for efficiently automating work processes in industrial facilities. The user sends work documents to a server via a terminal. The server analyzes the documents using a program with natural language processing technology and selects appropriate machine control tools. Specifically, the server analyzes the documents using a natural language processing toolkit (e.g., spaCy or NLTK) and, based on the analysis results, selects the optimal tool for automation (e.g., Ansible or Docker). Based on the selected tool, the server generates control code. This control code is applied to the execution environment (e.g., a specified PLC or SCADA system) to control machinery or robots.

[0294] The control codes generated in this way are applied in the specified environment to automate the work process. The server reports the execution results to the user, allowing them to monitor the progress of the work. Furthermore, by identifying procedures that are difficult to automate based on the analysis results and informing the user, it becomes possible to efficiently supplement necessary manual work.

[0295] As a concrete example, this automated system manages the process of preparing tools and positioning parts according to a schedule during the regular maintenance of robot arms used on an assembly line. Users can use prompt statements such as the following:

[0296] Example of a prompt:

[0297] "Automate the process of checking the oil level of the robotic arm and refilling it as needed."

[0298] "I would like you to analyze the setup instructions for the tools needed for the next product change and automate the preparation process."

[0299] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0300] Step 1:

[0301] The server receives a work document from the user. This received document is the data to be analyzed for natural language processing. The server converts this document into internal data and prepares it for use in the next analysis step.

[0302] Step 2:

[0303] The server analyzes the received work documents using a natural language processing toolkit (e.g., spaCy or NLTK). The data processing performed here involves extracting keywords and structural information from the documents. As a result of the analysis, the work procedures and important elements are extracted and organized into specific instructions.

[0304] Step 3:

[0305] The server selects the optimal machine control tool based on the analysis results. Specifically, it searches for libraries of corresponding automation tools (e.g., Ansible or Docker) based on the extracted keywords and determines the most suitable tool. The input is the analysis results, and the output is the selected control tool.

[0306] Step 4:

[0307] The server generates control code using the selected machine control tool. This generation process is an operation that executes the templates and scripts of the selected tool to generate specific control code. The input is the selected tool, and the output is the generated control code.

[0308] Step 5:

[0309] The server applies the generated control code to the target execution environment. Specifically, it transmits the code to the PLC or SCADA system in the industrial facility and performs a process to start the automatic operation of the device. The output is the operation result in the execution environment.

[0310] Step 6:

[0311] The server analyzes the execution result and reports it to the user. Specifically, based on the data obtained from the execution environment, it determines whether the operation was completed normally or there was an abnormality. The input is the execution result data, and the output is the content of the report.

[0312] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform specific processing using the user's emotion.

[0313] The present invention provides a system that realizes the selection and execution of an automation process from the analysis of a work instruction sheet, and further flexible response based on the user's emotional reaction by combining a natural language processing function and an emotion engine. This system operates centered around a server and aims to improve the user's operation efficiency and reduce the mental burden.

[0314] The user sends a work order to the server via their terminal. The server uses natural language processing to analyze the content of the work order and identify the scope of automation. Based on the analysis results, it then selects the optimal automation tool and generates the necessary automation code. The generated code is applied in the configured execution environment, and the work is performed automatically.

[0315] In addition, the server includes an emotion engine that measures the user's emotional state while working and uses that information to further optimize the process. Specifically, it provides a more comfortable working environment by suggesting ways to simplify processes or increasing support when the user is feeling stressed. For example, when automating the email server setup process, the system enhances support by providing additional explanations and generating more detailed instructions if the user expresses anxiety.

[0316] In this way, the present invention can integrate analyzed data with emotional feedback to improve work efficiency and user experience. As a result, it achieves improved automation accuracy and quality of work procedures, reducing user errors and burdens.

[0317] The following describes the processing flow.

[0318] Step 1:

[0319] The user uploads the work order to the server using their terminal. The server starts operating in response to this action, receiving and saving the work order file.

[0320] Step 2:

[0321] The server activates its natural language processing function to analyze the contents of the work instructions. The analysis extracts important keywords and operational steps, and categorizes the procedures based on these.

[0322] Step 3:

[0323] Based on the analysis results, the server selects the most suitable automation tool. Depending on the nature and scope of the procedure, it selects tools such as those for system configuration changes or test automation.

[0324] Step 4:

[0325] The server generates automation code corresponding to the selected automation tool. The generated code includes specific operational instructions based on the work order and is formatted into an executable form.

[0326] Step 5:

[0327] The server applies the automated code generated in the execution environment. Before application, it verifies that it works correctly in the test environment, and if there are no problems, it is deployed to the production environment.

[0328] Step 6:

[0329] The server uses an emotion engine to monitor the user's emotional state while they are working. While the user interacts with the interface, it analyzes camera footage and input patterns to infer their feelings.

[0330] Step 7:

[0331] Based on the analysis results of the emotion engine, the server provides support tailored to the user. For example, if the user is feeling frustrated, it may present a tutorial or simplify an automated process.

[0332] Step 8:

[0333] After all processes are complete, the server generates a report of the execution results and provides it to the user. Furthermore, it updates the work procedure manual based on user feedback to improve future operations.

[0334] (Example 2)

[0335] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0336] Conventional automation technologies were limited to analyzing work instructions and generating and executing automation code, and were unable to respond flexibly while considering the user's emotional state. This could lead to user stress, making it difficult to improve work efficiency and usability. Furthermore, decisions regarding parts that were difficult to automate and updates to procedures based on execution results were often performed manually, resulting in a significant burden.

[0337] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0338] In this invention, the server includes means for analyzing the contents of an information instruction document, means for selecting an appropriate automation device based on the analyzed information instruction document, means for generating an automation program for the selected automation device, and means for evaluating the user's emotional state and making suggestions to optimize the work process based on that evaluation. This enables flexible automation that responds to the user's emotions, thereby improving work efficiency and usability.

[0339] "Natural language processing" refers to the technology used by computers to understand and analyze human language and extract its meaning.

[0340] An "information instruction sheet" is a document that contains instructions and details necessary for a task or process, written in natural language.

[0341] "Automated equipment" is a general term for hardware or software that automatically performs specified tasks according to a program.

[0342] An "automation program" is software code or a script designed to perform a specific task without human intervention.

[0343] The "implementation environment" refers to the environment, including the computer resources and system settings necessary for the automation program to run.

[0344] "Assessing a user's emotional state" is the process of detecting a user's emotional response and quantifying or categorizing that state.

[0345] "Suggestions for optimizing the work process" refer to improvement measures or alternative procedures instructed by the system to enhance the user's work efficiency and comfort.

[0346] This invention combines natural language processing capabilities with an emotion evaluation engine to provide analysis of work instructions, selection and execution of automated processes, and adaptive support based on user emotions. It operates server-centric, and the overall process is carried out as follows:

[0347] The user first sends a work instruction sheet to the server via their terminal. Since this instruction sheet is written in natural language, it can be used by users without special technical knowledge. The server receives the instruction sheet and analyzes it using natural language processing capabilities. For this analysis, a general-purpose machine learning library, for example, is used as the natural language processing library.

[0348] The server selects the appropriate automation device based on the analysis results. During the selection process, commonly used automation software, such as RPA solutions, may be utilized. Subsequently, an automation program is generated for the selected tool. This program is automatically created in script format and applied in the configured execution environment.

[0349] Furthermore, the server is equipped with technology for evaluating user emotions. This includes techniques such as facial recognition and voice analysis, and in some cases, a general-purpose emotion analysis API. If the server determines that a user is experiencing stress or anxiety during their work, it provides additional instructions and support. For example, it might display detailed guides in the user interface or provide supportive video instructions.

[0350] As a concrete example, suppose a user sends a work order to the server stating, "Automatically create the folder structure for a new project." In this case, the server analyzes the order, selects the appropriate folder generation program, and executes it in the configured environment. If the user expresses any concerns along the way, the server displays detailed instructions on the screen to help the user understand.

[0351] An example of a prompt message might be: "Please automate the setup process for new mail servers. Also, please provide detailed instructions for users who may feel unsure."

[0352] This embodiment of the invention makes it possible to realize an automated work process that is easy for users to operate and reduces their mental burden.

[0353] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0354] Step 1:

[0355] The user inputs a work instruction document written in natural language using a terminal and sends it to the server. This input is a document clearly describing the specific work to be done. The user's actions include creating and sending the instruction document. The server receives this instruction document and proceeds to the next analysis step.

[0356] Step 2:

[0357] The server analyzes received work orders using natural language processing capabilities. The input is the work order received from the user, and the output is the analyzed specific task information. Specific data processing includes text segmentation, keyword extraction, and contextual understanding. Based on the analysis results, the server determines which parts can be automated.

[0358] Step 3:

[0359] The server selects the most suitable automation device based on the analysis results. It receives the analyzed task information as input and the selected automation device as output. In this step, it searches the database for corresponding automation tools and scripts and selects the tool that can perform the task most efficiently.

[0360] Step 4:

[0361] The server generates automation programs for the selected automation devices. It takes the selection results as input and generates specific program code as output. Data calculations include script assembly and automatic generation of API calls.

[0362] Step 5:

[0363] The generated automation program is executed in the implementation environment. The server applies the program code to the implementation environment and automatically performs the configured tasks. The output is the result of the automated task execution, and the results are recorded in a predetermined format.

[0364] Step 6:

[0365] The server evaluates the user's emotional state during operation. Input is data such as the user's facial expressions and voice, and output is the evaluation result of the emotional state. Specific operations include real-time data capture and analysis.

[0366] Step 7:

[0367] Based on the user's emotional assessment, the server proposes optimizations to the work process. The emotional state assessment is taken as input, and the output is the optimized proposal. For example, if the server determines that the work is complex and stressful, it will offer suggestions to simplify the work procedure.

[0368] (Application Example 2)

[0369] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0370] In modern manufacturing, while there is a demand for automating processes based on work instructions, the challenge lies in reducing the mental burden and stress on workers and enabling them to perform tasks efficiently and flexibly. In particular, when workers feel anxious about complex instructions, there is a lack of appropriate procedure optimization and emotionally responsive support, and this problem needs to be addressed.

[0371] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0372] In this invention, the server includes means for analyzing the contents of work instructions, means for selecting an appropriate automation tool based on the analyzed work instructions, means for generating automation code for the selected automation tool, and means for acquiring emotional states and optimizing work procedures based on that information. This enables the selection and execution of automation processes from the analysis of work instructions, and further enables the provision of appropriate support according to the emotional state of the worker, thereby realizing an efficient and less stressful work environment.

[0373] "Natural language processing" refers to technologies that analyze input text data and understand its meaning and intent.

[0374] A "work instruction sheet" is a document that describes the procedures and conditions necessary to perform a specific task.

[0375] An "automation tool" is software or a system used to efficiently perform a specific task.

[0376] "Automation code" is a program written to execute a process automatically.

[0377] The "execution environment" refers to the hardware and software configuration required for a program to run.

[0378] "Emotional state" refers to a user's psychological or emotional condition, and measuring its changes is an indicator that can be used to support their work.

[0379] "Optimization" refers to improving existing processes and procedures to achieve a specific objective and maximize efficiency.

[0380] A system for implementing this invention consists of a server, a user terminal, and a work environment.

[0381] Program Overview

[0382] The server executes specific libraries (e.g., spaCy and Transformers) to perform natural language processing and analyzes work instructions received from the user's terminal. Based on the analysis results, it selects appropriate automation tools and generates program code. This automation code can then be run in an execution environment built on the server. Furthermore, the server uses an emotion analysis engine (e.g., IBM Watson) to evaluate the user's emotional state in real time and optimize the work procedure accordingly. For example, if the user is feeling anxious, it provides additional explanations or support.

[0383] Hardware and software used

[0384] Server: This is the central unit that analyzes work instructions and selects automation tools.

[0385] User terminal (e.g., smart glasses): A device used by workers to send work instructions to the server.

[0386] Emotion analysis engine: This is software used to measure a user's emotional state.

[0387] Specific example

[0388] Consider a scenario on a manufacturing line where a user puts on smart glasses and begins the setup procedure for a new machine. The user takes a picture of the work instructions through the glasses and sends them to a server. The server analyzes the information, generates the necessary automation code, and executes it. If the emotion engine detects the user's anxiety, it provides more specific setup instructions or visual guides to the glasses to aid understanding.

[0389] Example of a prompt

[0390] "Please explain the assembly procedure for the new parts. If there are any parts of the procedure that are complex or difficult to understand, please provide a detailed explanation."

[0391] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0392] Step 1:

[0393] The user uses smart glasses to photograph the work instruction sheet. At this time, the user captures the image of the work instruction sheet into their device and sends it to the server as digital data. The input is the image of the work instruction sheet, which forms the basis for subsequent data processing.

[0394] Step 2:

[0395] The server applies OCR technology to the received image of the work instruction and converts it into text data. This is data processing to convert image data into string data. The output is the text data of the work instruction.

[0396] Step 3:

[0397] The server uses a natural language processing library to analyze the text data of the work instructions in detail. This involves semantic and syntactic analysis of the text to identify tasks that can be automated. The input to this process is text data obtained by OCR, and the output is information about the analyzed tasks.

[0398] Step 4:

[0399] Based on the analyzed information, the server selects the optimal automation tool and generates the necessary automation code. The input is the task information obtained in step 3, and the output code is associated with a specific tool. At this stage, the server performs data calculations and creates code aimed at efficient task execution.

[0400] Step 5:

[0401] The generated automation code is run in a specific execution environment on the server. Here, the code is applied and its results are retrieved. The output of this process is the execution results and their report. The server monitors the execution process and collects results as needed.

[0402] Step 6:

[0403] The server uses an emotion analysis engine to monitor and evaluate the user's emotional state in real time. It analyzes the user's stress level and provides simplified procedures or additional explanations as needed. Inputs include the user's biometric data and facial expressions, and the analysis results are the output.

[0404] Step 7:

[0405] The user reviews the optimized instructions displayed on the smart glasses and proceeds with the task. Based on the sentiment analysis results obtained in Step 6, the server provides improvement suggestions, and the user utilizes specific work support. At this stage, the outputted instructions are presented in a way that is appropriate to the work environment.

[0406] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0407] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0408] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0409] [Third Embodiment]

[0410] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0411] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0412] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0413] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0414] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0415] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0416] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0417] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0418] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0419] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0420] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0421] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0422] This invention provides a system for performing the analysis and automation of work instructions, which were previously done manually. In implementing the invention, a server primarily plays a central role, analyzing work instructions using natural language processing technology and selecting the optimal automation tool to improve work efficiency.

[0423] Specifically, when a user uploads a work order to the server via their terminal, the server analyzes its contents using natural language processing technology. Based on the analysis results, the server selects the most suitable automation tool for the procedure and generates automation code corresponding to that tool. This code is then applied to the execution environment according to the system's instructions and used to automate the operations required in the work order.

[0424] As a concrete example of this system, consider a scenario where a user sends a "database backup procedure document" to the server. The server analyzes the document and selects an appropriate automation tool (such as a scripting tool or configuration management tool) to perform a safe and efficient backup operation on the database. Based on the selected tool, the server generates an appropriate script, automates the backup process, and reports the results to the user.

[0425] This simplifies complex and time-consuming tasks that were previously performed manually, leading to increased efficiency and improved quality. Furthermore, the server documents the results of each step and automatically generates work procedure documents, providing reference materials for subsequent tasks. This allows users to reduce management costs while enabling effective business operations.

[0426] The following describes the processing flow.

[0427] Step 1:

[0428] The user uploads the work order to the server using a terminal. The server receives the uploaded file and confirms its storage location.

[0429] Step 2:

[0430] The server activates its natural language processing function and analyzes the contents of the received work instructions. The analysis involves breaking down the procedures, extracting keywords, and identifying related work categories.

[0431] Step 3:

[0432] Based on the analysis results, the server selects the appropriate automation tool. For example, if scripting is required, it selects the appropriate scripting tool.

[0433] Step 4:

[0434] The server automatically generates code or scripts corresponding to the selected automation tool. This generated code includes content corresponding to each step of the work order.

[0435] Step 5:

[0436] The server applies the generated code to a specific execution environment. Before application, it runs the code in a test environment to check for errors, and if there are no problems, it is deployed to the production environment.

[0437] Step 6:

[0438] The server collects the execution results and provides them to the user as a report. It also automatically generates updated work procedures based on the results, preparing for future tasks.

[0439] Step 7:

[0440] The server identifies tasks that are difficult to automate and presents them to the user as a list. The user can then use this list to proceed with the tasks manually.

[0441] (Example 1)

[0442] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0443] In the process of analyzing and automating work instructions, there is a problem in quickly and accurately determining which procedures can be automated and which tools are best suited for that process. Furthermore, the process of identifying steps that require manual work and efficiently notifying the responsible personnel is cumbersome. In addition, after the execution of the generated automation scripts, properly recording the updated information reflecting the results and creating documentation to be used for subsequent tasks requires significant resources.

[0444] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0445] In this invention, the server includes means for receiving work instructions with natural language processing capabilities, means for analyzing the received work instructions to identify key procedures, and means for selecting the optimal automation software based on the analysis results. This enables improved analysis accuracy and a more efficient selection process. Furthermore, by detecting manual processes and quickly notifying the responsible personnel, overall work efficiency is improved. In addition, by generating immediately updated work instruction documents based on execution results, the effort required for document creation is reduced, and the overall quality of operations is improved.

[0446] "Natural language processing" is a technology that converts written text into a format that a computer can understand, and then analyzes and processes the information contained within it.

[0447] A "work instruction sheet" is a document that details how to perform a specific task or procedure.

[0448] "Automation software" is a program that converts tasks performed by humans into tasks that can be executed by machines, thereby automating business processes.

[0449] An "automation script" is a set of instructions or code created to automatically execute specific procedures or tasks.

[0450] "Operational infrastructure" refers to the system environment and infrastructure necessary for the normal execution and management of programs and scripts.

[0451] A "manual process" is a business process that requires direct human involvement because automation is difficult or impossible.

[0452] A "work instruction document" is a document that outlines the specific procedures and policies for carrying out a task, and is used as a reference for subsequent processes and related work.

[0453] This invention is a system for efficiently analyzing and automating work instructions. Its main components include a terminal, a server, a natural language processing model, and various automation software.

[0454] First, the user uploads the work order to the server using a terminal. The terminal consists of a typical computer or tablet device. In this process, the user selects files and sends them via a browser.

[0455] The server analyzes the received work instructions using a natural language processing model. Specific software used includes generative AI models such as BERT and GPT. The server tokenizes the text and extracts important work procedures and information by understanding the context.

[0456] Based on the analysis results, the server selects the most suitable automation software. For example, Ansible or Puppet, which are suitable for operational use, might be selected. After selection, the server generates an automation script tailored to the identified task. This includes formatting the script's syntax and structure for the chosen software.

[0457] The generated script is executed on the server's operational infrastructure. The execution results are recorded as logs and reported to the user. Furthermore, documentation is created based on the execution results, and a work instruction document that can be used as a reference for subsequent tasks is automatically generated.

[0458] For example, if a user sends a "database backup procedure document" to the server, the server analyzes its contents and selects and generates appropriate tools and scripts to ensure safe and efficient data backup. The generated scripts are executed on the server, and after successful completion, the results are reported to the user.

[0459] To use this system, you can input a prompt like the following into the generating AI model: "Analyze this text and suggest the best tools and scripts for automation." This allows the system to provide specific information to support decisions regarding automation.

[0460] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0461] Step 1:

[0462] The user uploads a work order to the server using a terminal. The input is the work order file selected by the user from the terminal. On the terminal, the user selects the target work order from a file selection dialog and clicks the send button. The output is the work order saved on the server. The server stores this file in a specified directory for analysis preparation.

[0463] Step 2:

[0464] The server analyzes the content of the uploaded work instructions using natural language processing technology. The input is the work instructions stored on the server in Step 1. The server uses a generative AI model to tokenize the text, understand the context, and perform data processing to extract important procedures and keywords. The output is structured data of the analyzed work instructions. This data is used in subsequent processing.

[0465] Step 3:

[0466] The server selects the optimal automation software based on the analysis results. The input is the analysis result data from step 2. The server performs a data calculation to select the most suitable automation tool from a list of automation tools according to the specified conditions. The output is the name of the selected automation software. This selection result then leads to script generation.

[0467] Step 4:

[0468] The server generates an automation script based on the selected automation software. The inputs are the names of the automation tools selected in step 3 and the analysis results from step 2. The server performs data processing to generate the script by combining the appropriate script syntax and structure. The output is the generated script file. This script then proceeds to the next execution process.

[0469] Step 5:

[0470] The server executes the generated script on the production platform. The input is the script generated in step 4. During execution, the server records operation logs and checks the success or failure of each step. The output is the execution result log data. This log is used for reporting work results and generating documentation.

[0471] Step 6:

[0472] The server automatically generates documents based on the execution results and reports them to the user. The input is the execution result log obtained in step 5. The server analyzes the execution results and processes the data to format it into a document. The output is the generated report and updated work instruction document. These are provided to the user and contribute to improving work efficiency.

[0473] (Application Example 1)

[0474] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0475] In modern industrial facilities, complex and diverse work processes exist, making efficient management and automation extremely difficult. In particular, the process of extracting appropriate procedures from work documents and implementing automation relies on manual work, which is time-consuming and labor-intensive. Furthermore, when procedures that are difficult to automate must be manually verified, the overall efficiency suffers. There is a need to solve these problems and provide a system that effectively automates work processes in industrial facilities while also enabling rapid responses to procedures that are difficult to automate.

[0476] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0477] In this invention, the server includes means for analyzing the content of work documents, means for selecting appropriate machine control tools based on the analyzed work documents, and means for generating control codes for the selected machine control tools. This enables the automation of industrial facility functions and the effective management of work processes. Furthermore, by extracting procedures that are difficult to automate based on the analysis results and presenting them to the person in charge, it enables the rapid and efficient supplementation of necessary manual work.

[0478] "Natural language processing" refers to the technology that allows computers to understand and process human language, enabling document analysis and information extraction.

[0479] A "work document" is a document that contains the procedures and instructions necessary to perform a specific task, and serves as a guide for efficient work in industrial facilities.

[0480] "Analysis" is the act of extracting structure and meaning from specific data or information and performing processing to gain a deeper understanding.

[0481] A "machine control tool" is a software tool used to efficiently and automatically operate and manage equipment and machinery within industrial facilities.

[0482] A "control code" is a set of instructions and programs necessary to operate a specific machine or device, and is a crucial element in realizing machine control.

[0483] The "execution environment" refers to the combination of hardware and software required for the generated control code to actually run.

[0484] A description of the embodiment for carrying out the invention will be provided.

[0485] This invention provides a system for efficiently automating work processes in industrial facilities. The user sends work documents to a server via a terminal. The server analyzes the documents using a program with natural language processing technology and selects appropriate machine control tools. Specifically, the server analyzes the documents using a natural language processing toolkit (e.g., spaCy or NLTK) and, based on the analysis results, selects the optimal tool for automation (e.g., Ansible or Docker). Based on the selected tool, the server generates control code. This control code is applied to the execution environment (e.g., a specified PLC or SCADA system) to control machinery or robots.

[0486] The control codes generated in this way are applied in the specified environment to automate the work process. The server reports the execution results to the user, allowing them to monitor the progress of the work. Furthermore, by identifying procedures that are difficult to automate based on the analysis results and informing the user, it becomes possible to efficiently supplement necessary manual work.

[0487] As a concrete example, this automated system manages the process of preparing tools and positioning parts according to a schedule during the regular maintenance of robot arms used on an assembly line. Users can use prompt statements such as the following:

[0488] Example of a prompt:

[0489] "Automate the process of checking the oil level of the robotic arm and refilling it as needed."

[0490] "I would like you to analyze the setup instructions for the tools needed for the next product change and automate the preparation process."

[0491] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0492] Step 1:

[0493] The server receives a work document from the user. This received document is the data to be analyzed for natural language processing. The server converts this document into internal data and prepares it for use in the next analysis step.

[0494] Step 2:

[0495] The server analyzes the received work documents using a natural language processing toolkit (e.g., spaCy or NLTK). The data processing performed here involves extracting keywords and structural information from the documents. As a result of the analysis, the work procedures and important elements are extracted and organized into specific instructions.

[0496] Step 3:

[0497] The server selects the optimal machine control tool based on the analysis results. Specifically, it searches for libraries of corresponding automation tools (e.g., Ansible or Docker) based on the extracted keywords and determines the most suitable tool. The input is the analysis results, and the output is the selected control tool.

[0498] Step 4:

[0499] The server generates control codes using the selected machine control tool. This generation process executes templates and scripts for the selected tool to generate specific control codes. The input is the selected tool, and the output is the generated control code.

[0500] Step 5:

[0501] The server applies the generated control code to the target execution environment. Specifically, it sends the code to the PLC or SCADA system of the industrial facility and performs the processing necessary to start the automated operation of the equipment. The output is the result of the operation in the execution environment.

[0502] Step 6:

[0503] The server analyzes the execution results and reports them to the user. Specifically, it determines whether the operation was completed successfully or if there were any errors based on the data obtained from the execution environment. The input is the execution result data, and the output is the report content.

[0504] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0505] This invention provides a system that combines natural language processing capabilities with an emotion engine to enable analysis of work instructions, selection and execution of automated processes, and flexible responses based on the user's emotional reactions. This system operates server-centric, aiming to improve user operational efficiency and reduce mental burden.

[0506] The user sends a work order to the server via their terminal. The server uses natural language processing to analyze the content of the work order and identify the scope of automation. Based on the analysis results, it then selects the optimal automation tool and generates the necessary automation code. The generated code is applied in the configured execution environment, and the work is performed automatically.

[0507] In addition, the server includes an emotion engine that measures the user's emotional state while working and uses that information to further optimize the process. Specifically, it provides a more comfortable working environment by suggesting ways to simplify processes or increasing support when the user is feeling stressed. For example, when automating the email server setup process, the system enhances support by providing additional explanations and generating more detailed instructions if the user expresses anxiety.

[0508] In this way, the present invention can integrate analyzed data with emotional feedback to improve work efficiency and user experience. As a result, it achieves improved automation accuracy and quality of work procedures, reducing user errors and burdens.

[0509] The following describes the processing flow.

[0510] Step 1:

[0511] The user uploads the work order to the server using their terminal. The server starts operating in response to this action, receiving and saving the work order file.

[0512] Step 2:

[0513] The server activates its natural language processing function to analyze the contents of the work instructions. The analysis extracts important keywords and operational steps, and categorizes the procedures based on these.

[0514] Step 3:

[0515] Based on the analysis results, the server selects the most suitable automation tool. Depending on the nature and scope of the procedure, it selects tools such as those for system configuration changes or test automation.

[0516] Step 4:

[0517] The server generates automation code corresponding to the selected automation tool. The generated code includes specific operational instructions based on the work order and is formatted into an executable form.

[0518] Step 5:

[0519] The server applies the automated code generated in the execution environment. Before application, it verifies that it works correctly in the test environment, and if there are no problems, it is deployed to the production environment.

[0520] Step 6:

[0521] The server uses an emotion engine to monitor the user's emotional state while they are working. While the user interacts with the interface, it analyzes camera footage and input patterns to infer their feelings.

[0522] Step 7:

[0523] Based on the analysis results of the emotion engine, the server provides support tailored to the user. For example, if the user is feeling frustrated, it may present a tutorial or simplify an automated process.

[0524] Step 8:

[0525] After all processes are complete, the server generates a report of the execution results and provides it to the user. Furthermore, it updates the work procedure manual based on user feedback to improve future operations.

[0526] (Example 2)

[0527] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0528] Conventional automation technologies were limited to analyzing work instructions and generating and executing automation code, and were unable to respond flexibly while considering the user's emotional state. This could lead to user stress, making it difficult to improve work efficiency and usability. Furthermore, decisions regarding parts that were difficult to automate and updates to procedures based on execution results were often performed manually, resulting in a significant burden.

[0529] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0530] In this invention, the server includes means for analyzing the contents of an information instruction document, means for selecting an appropriate automation device based on the analyzed information instruction document, means for generating an automation program for the selected automation device, and means for evaluating the user's emotional state and making suggestions to optimize the work process based on that evaluation. This enables flexible automation that responds to the user's emotions, thereby improving work efficiency and usability.

[0531] "Natural language processing" refers to the technology used by computers to understand and analyze human language and extract its meaning.

[0532] An "information instruction sheet" is a document that contains instructions and details necessary for a task or process, written in natural language.

[0533] "Automated equipment" is a general term for hardware or software that automatically performs specified tasks according to a program.

[0534] An "automation program" is software code or a script designed to perform a specific task without human intervention.

[0535] The "implementation environment" refers to the environment, including the computer resources and system settings necessary for the automation program to run.

[0536] "Assessing a user's emotional state" is the process of detecting a user's emotional response and quantifying or categorizing that state.

[0537] "Suggestions for optimizing the work process" refer to improvement measures or alternative procedures instructed by the system to enhance the user's work efficiency and comfort.

[0538] This invention combines natural language processing capabilities with an emotion evaluation engine to provide analysis of work instructions, selection and execution of automated processes, and adaptive support based on user emotions. It operates server-centric, and the overall process is carried out as follows:

[0539] The user first sends a work instruction sheet to the server via their terminal. Since this instruction sheet is written in natural language, it can be used by users without special technical knowledge. The server receives the instruction sheet and analyzes it using natural language processing capabilities. For this analysis, a general-purpose machine learning library, for example, is used as the natural language processing library.

[0540] The server selects the appropriate automation device based on the analysis results. During the selection process, commonly used automation software, such as RPA solutions, may be utilized. Subsequently, an automation program is generated for the selected tool. This program is automatically created in script format and applied in the configured execution environment.

[0541] Furthermore, the server is equipped with technology for evaluating user emotions. This includes techniques such as facial recognition and voice analysis, and in some cases, a general-purpose emotion analysis API. If the server determines that a user is experiencing stress or anxiety during their work, it provides additional instructions and support. For example, it might display detailed guides in the user interface or provide supportive video instructions.

[0542] As a concrete example, suppose a user sends a work order to the server stating, "Automatically create the folder structure for a new project." In this case, the server analyzes the order, selects the appropriate folder generation program, and executes it in the configured environment. If the user expresses any concerns along the way, the server displays detailed instructions on the screen to help the user understand.

[0543] An example of a prompt message might be: "Please automate the setup process for new mail servers. Also, please provide detailed instructions for users who may feel unsure."

[0544] This embodiment of the invention makes it possible to realize an automated work process that is easy for users to operate and reduces their mental burden.

[0545] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0546] Step 1:

[0547] The user inputs a work instruction document written in natural language using a terminal and sends it to the server. This input is a document clearly describing the specific work to be done. The user's actions include creating and sending the instruction document. The server receives this instruction document and proceeds to the next analysis step.

[0548] Step 2:

[0549] The server analyzes received work orders using natural language processing capabilities. The input is the work order received from the user, and the output is the analyzed specific task information. Specific data processing includes text segmentation, keyword extraction, and contextual understanding. Based on the analysis results, the server determines which parts can be automated.

[0550] Step 3:

[0551] The server selects the most suitable automation device based on the analysis results. It receives the analyzed task information as input and the selected automation device as output. In this step, it searches the database for corresponding automation tools and scripts and selects the tool that can perform the task most efficiently.

[0552] Step 4:

[0553] The server generates automation programs for the selected automation devices. It takes the selection results as input and generates specific program code as output. Data calculations include script assembly and automatic generation of API calls.

[0554] Step 5:

[0555] The generated automation program is executed in the implementation environment. The server applies the program code to the implementation environment and automatically performs the configured tasks. The output is the result of the automated task execution, and the results are recorded in a predetermined format.

[0556] Step 6:

[0557] The server evaluates the user's emotional state during operation. Input is data such as the user's facial expressions and voice, and output is the evaluation result of the emotional state. Specific operations include real-time data capture and analysis.

[0558] Step 7:

[0559] Based on the user's emotional assessment, the server proposes optimizations to the work process. The emotional state assessment is taken as input, and the output is the optimized proposal. For example, if the server determines that the work is complex and stressful, it will offer suggestions to simplify the work procedure.

[0560] (Application Example 2)

[0561] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0562] In modern manufacturing, while there is a demand for automating processes based on work instructions, the challenge lies in reducing the mental burden and stress on workers and enabling them to perform tasks efficiently and flexibly. In particular, when workers feel anxious about complex instructions, there is a lack of appropriate procedure optimization and emotionally responsive support, and this problem needs to be addressed.

[0563] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0564] In this invention, the server includes means for analyzing the contents of work instructions, means for selecting an appropriate automation tool based on the analyzed work instructions, means for generating automation code for the selected automation tool, and means for acquiring emotional states and optimizing work procedures based on that information. This enables the selection and execution of automation processes from the analysis of work instructions, and further enables the provision of appropriate support according to the emotional state of the worker, thereby realizing an efficient and less stressful work environment.

[0565] "Natural language processing" refers to technologies that analyze input text data and understand its meaning and intent.

[0566] A "work instruction sheet" is a document that describes the procedures and conditions necessary to perform a specific task.

[0567] An "automation tool" is software or a system used to efficiently perform a specific task.

[0568] "Automation code" is a program written to execute a process automatically.

[0569] The "execution environment" refers to the hardware and software configuration required for a program to run.

[0570] "Emotional state" refers to a user's psychological or emotional condition, and measuring its changes is an indicator that can be used to support their work.

[0571] "Optimization" refers to improving existing processes and procedures to achieve a specific objective and maximize efficiency.

[0572] A system for implementing this invention consists of a server, a user terminal, and a work environment.

[0573] Program Overview

[0574] The server executes specific libraries (e.g., spaCy and Transformers) to perform natural language processing and analyzes work instructions received from the user's terminal. Based on the analysis results, it selects appropriate automation tools and generates program code. This automation code can then be run in an execution environment built on the server. Furthermore, the server uses an emotion analysis engine (e.g., IBM Watson) to evaluate the user's emotional state in real time and optimize the work procedure accordingly. For example, if the user is feeling anxious, it provides additional explanations or support.

[0575] Hardware and software used

[0576] Server: This is the central unit that analyzes work instructions and selects automation tools.

[0577] User terminal (e.g., smart glasses): A device used by workers to send work instructions to the server.

[0578] Emotion analysis engine: This is software used to measure a user's emotional state.

[0579] Specific example

[0580] Consider a scenario on a manufacturing line where a user puts on smart glasses and begins the setup procedure for a new machine. The user takes a picture of the work instructions through the glasses and sends them to a server. The server analyzes the information, generates the necessary automation code, and executes it. If the emotion engine detects the user's anxiety, it provides more specific setup instructions or visual guides to the glasses to aid understanding.

[0581] Example of a prompt

[0582] "Please explain the assembly procedure for the new parts. If there are any parts of the procedure that are complex or difficult to understand, please provide a detailed explanation."

[0583] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0584] Step 1:

[0585] The user uses smart glasses to photograph the work instruction sheet. At this time, the user captures the image of the work instruction sheet into their device and sends it to the server as digital data. The input is the image of the work instruction sheet, which forms the basis for subsequent data processing.

[0586] Step 2:

[0587] The server applies OCR technology to the received image of the work instruction and converts it into text data. This is data processing to convert image data into string data. The output is the text data of the work instruction.

[0588] Step 3:

[0589] The server uses a natural language processing library to analyze the text data of the work instructions in detail. This involves semantic and syntactic analysis of the text to identify tasks that can be automated. The input to this process is text data obtained by OCR, and the output is information about the analyzed tasks.

[0590] Step 4:

[0591] Based on the analyzed information, the server selects the optimal automation tool and generates the necessary automation code. The input is the task information obtained in step 3, and the output code is associated with a specific tool. At this stage, the server performs data calculations and creates code aimed at efficient task execution.

[0592] Step 5:

[0593] The generated automation code is run in a specific execution environment on the server. Here, the code is applied and its results are retrieved. The output of this process is the execution results and their report. The server monitors the execution process and collects results as needed.

[0594] Step 6:

[0595] The server uses an emotion analysis engine to monitor and evaluate the user's emotional state in real time. It analyzes the user's stress level and provides simplified procedures or additional explanations as needed. Inputs include the user's biometric data and facial expressions, and the analysis results are the output.

[0596] Step 7:

[0597] The user reviews the optimized instructions displayed on the smart glasses and proceeds with the task. Based on the sentiment analysis results obtained in Step 6, the server provides improvement suggestions, and the user utilizes specific work support. At this stage, the outputted instructions are presented in a way that is appropriate to the work environment.

[0598] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0599] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0600] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0601] [Fourth Embodiment]

[0602] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0603] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0604] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0605] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0606] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0607] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0608] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0609] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0610] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0611] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0612] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0613] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0614] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0615] This invention provides a system for performing the analysis and automation of work instructions, which were previously done manually. In implementing the invention, a server primarily plays a central role, analyzing work instructions using natural language processing technology and selecting the optimal automation tool to improve work efficiency.

[0616] Specifically, when a user uploads a work order to the server via their terminal, the server analyzes its contents using natural language processing technology. Based on the analysis results, the server selects the most suitable automation tool for the procedure and generates automation code corresponding to that tool. This code is then applied to the execution environment according to the system's instructions and used to automate the operations required in the work order.

[0617] As a concrete example of this system, consider a scenario where a user sends a "database backup procedure document" to the server. The server analyzes the document and selects an appropriate automation tool (such as a scripting tool or configuration management tool) to perform a safe and efficient backup operation on the database. Based on the selected tool, the server generates an appropriate script, automates the backup process, and reports the results to the user.

[0618] This simplifies complex and time-consuming tasks that were previously performed manually, leading to increased efficiency and improved quality. Furthermore, the server documents the results of each step and automatically generates work procedure documents, providing reference materials for subsequent tasks. This allows users to reduce management costs while enabling effective business operations.

[0619] The following describes the processing flow.

[0620] Step 1:

[0621] The user uploads the work order to the server using a terminal. The server receives the uploaded file and confirms its storage location.

[0622] Step 2:

[0623] The server activates its natural language processing function and analyzes the contents of the received work instructions. The analysis involves breaking down the procedures, extracting keywords, and identifying related work categories.

[0624] Step 3:

[0625] Based on the analysis results, the server selects the appropriate automation tool. For example, if scripting is required, it selects the appropriate scripting tool.

[0626] Step 4:

[0627] The server automatically generates code or scripts corresponding to the selected automation tool. This generated code includes content corresponding to each step of the work order.

[0628] Step 5:

[0629] The server applies the generated code to a specific execution environment. Before application, it runs the code in a test environment to check for errors, and if there are no problems, it is deployed to the production environment.

[0630] Step 6:

[0631] The server collects the execution results and provides them to the user as a report. It also automatically generates updated work procedures based on the results, preparing for future tasks.

[0632] Step 7:

[0633] The server identifies tasks that are difficult to automate and presents them to the user as a list. The user can then use this list to proceed with the tasks manually.

[0634] (Example 1)

[0635] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0636] In the process of analyzing and automating work instructions, there is a problem in quickly and accurately determining which procedures can be automated and which tools are best suited for that process. Furthermore, the process of identifying steps that require manual work and efficiently notifying the responsible personnel is cumbersome. In addition, after the execution of the generated automation scripts, properly recording the updated information reflecting the results and creating documentation to be used for subsequent tasks requires significant resources.

[0637] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0638] In this invention, the server includes means for receiving work instructions with natural language processing capabilities, means for analyzing the received work instructions to identify key procedures, and means for selecting the optimal automation software based on the analysis results. This enables improved analysis accuracy and a more efficient selection process. Furthermore, by detecting manual processes and quickly notifying the responsible personnel, overall work efficiency is improved. In addition, by generating immediately updated work instruction documents based on execution results, the effort required for document creation is reduced, and the overall quality of operations is improved.

[0639] "Natural language processing" is a technology that converts written text into a format that a computer can understand, and then analyzes and processes the information contained within it.

[0640] A "work instruction sheet" is a document that details how to perform a specific task or procedure.

[0641] "Automation software" is a program that converts tasks performed by humans into tasks that can be executed by machines, thereby automating business processes.

[0642] An "automation script" is a set of instructions or code created to automatically execute specific procedures or tasks.

[0643] "Operational infrastructure" refers to the system environment and infrastructure necessary for the normal execution and management of programs and scripts.

[0644] A "manual process" is a business process that requires direct human involvement because automation is difficult or impossible.

[0645] A "work instruction document" is a document that outlines the specific procedures and policies for carrying out a task, and is used as a reference for subsequent processes and related work.

[0646] This invention is a system for efficiently analyzing and automating work instructions. Its main components include a terminal, a server, a natural language processing model, and various automation software.

[0647] First, the user uploads the work order to the server using a terminal. The terminal consists of a typical computer or tablet device. In this process, the user selects files and sends them via a browser.

[0648] The server analyzes the received work instructions using a natural language processing model. Specific software used includes generative AI models such as BERT and GPT. The server tokenizes the text and extracts important work procedures and information by understanding the context.

[0649] Based on the analysis results, the server selects the most suitable automation software. For example, Ansible or Puppet, which are suitable for operational use, might be selected. After selection, the server generates an automation script tailored to the identified task. This includes formatting the script's syntax and structure for the chosen software.

[0650] The generated script is executed on the server's operational infrastructure. The execution results are recorded as logs and reported to the user. Furthermore, documentation is created based on the execution results, and a work instruction document that can be used as a reference for subsequent tasks is automatically generated.

[0651] For example, if a user sends a "database backup procedure document" to the server, the server analyzes its contents and selects and generates appropriate tools and scripts to ensure safe and efficient data backup. The generated scripts are executed on the server, and after successful completion, the results are reported to the user.

[0652] To use this system, you can input a prompt like the following into the generating AI model: "Analyze this text and suggest the best tools and scripts for automation." This allows the system to provide specific information to support decisions regarding automation.

[0653] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0654] Step 1:

[0655] The user uploads a work order to the server using a terminal. The input is the work order file selected by the user from the terminal. On the terminal, the user selects the target work order from a file selection dialog and clicks the send button. The output is the work order saved on the server. The server stores this file in a specified directory for analysis preparation.

[0656] Step 2:

[0657] The server analyzes the content of the uploaded work instructions using natural language processing technology. The input is the work instructions stored on the server in Step 1. The server uses a generative AI model to tokenize the text, understand the context, and perform data processing to extract important procedures and keywords. The output is structured data of the analyzed work instructions. This data is used in subsequent processing.

[0658] Step 3:

[0659] The server selects the optimal automation software based on the analysis results. The input is the analysis result data from step 2. The server performs a data calculation to select the most suitable automation tool from a list of automation tools according to the specified conditions. The output is the name of the selected automation software. This selection result then leads to script generation.

[0660] Step 4:

[0661] The server generates an automation script based on the selected automation software. The inputs are the names of the automation tools selected in step 3 and the analysis results from step 2. The server performs data processing to generate the script by combining the appropriate script syntax and structure. The output is the generated script file. This script then proceeds to the next execution process.

[0662] Step 5:

[0663] The server executes the generated script on the production platform. The input is the script generated in step 4. During execution, the server records operation logs and checks the success or failure of each step. The output is the execution result log data. This log is used for reporting work results and generating documentation.

[0664] Step 6:

[0665] The server automatically generates documents based on the execution results and reports them to the user. The input is the execution result log obtained in step 5. The server analyzes the execution results and processes the data to format it into a document. The output is the generated report and updated work instruction document. These are provided to the user and contribute to improving work efficiency.

[0666] (Application Example 1)

[0667] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0668] In modern industrial facilities, complex and diverse work processes exist, making efficient management and automation extremely difficult. In particular, the process of extracting appropriate procedures from work documents and implementing automation relies on manual work, which is time-consuming and labor-intensive. Furthermore, when procedures that are difficult to automate must be manually verified, the overall efficiency suffers. There is a need to solve these problems and provide a system that effectively automates work processes in industrial facilities while also enabling rapid responses to procedures that are difficult to automate.

[0669] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0670] In this invention, the server includes means for analyzing the content of work documents, means for selecting appropriate machine control tools based on the analyzed work documents, and means for generating control codes for the selected machine control tools. This enables the automation of industrial facility functions and the effective management of work processes. Furthermore, by extracting procedures that are difficult to automate based on the analysis results and presenting them to the person in charge, it enables the rapid and efficient supplementation of necessary manual work.

[0671] "Natural language processing" refers to the technology that allows computers to understand and process human language, enabling document analysis and information extraction.

[0672] A "work document" is a document that contains the procedures and instructions necessary to perform a specific task, and serves as a guide for efficient work in industrial facilities.

[0673] "Analysis" is the act of extracting structure and meaning from specific data or information and performing processing to gain a deeper understanding.

[0674] A "machine control tool" is a software tool used to efficiently and automatically operate and manage equipment and machinery within industrial facilities.

[0675] A "control code" is a set of instructions and programs necessary to operate a specific machine or device, and is a crucial element in realizing machine control.

[0676] The "execution environment" refers to the combination of hardware and software required for the generated control code to actually run.

[0677] A description of the embodiment for carrying out the invention will be provided.

[0678] This invention provides a system for efficiently automating work processes in industrial facilities. The user sends work documents to a server via a terminal. The server analyzes the documents using a program with natural language processing technology and selects appropriate machine control tools. Specifically, the server analyzes the documents using a natural language processing toolkit (e.g., spaCy or NLTK) and, based on the analysis results, selects the optimal tool for automation (e.g., Ansible or Docker). Based on the selected tool, the server generates control code. This control code is applied to the execution environment (e.g., a specified PLC or SCADA system) to control machinery or robots.

[0679] The control codes generated in this way are applied in the specified environment to automate the work process. The server reports the execution results to the user, allowing them to monitor the progress of the work. Furthermore, by identifying procedures that are difficult to automate based on the analysis results and informing the user, it becomes possible to efficiently supplement necessary manual work.

[0680] As a concrete example, this automated system manages the process of preparing tools and positioning parts according to a schedule during the regular maintenance of robot arms used on an assembly line. Users can use prompt statements such as the following:

[0681] Example of a prompt:

[0682] "Automate the process of checking the oil level of the robotic arm and refilling it as needed."

[0683] "I would like you to analyze the setup instructions for the tools needed for the next product change and automate the preparation process."

[0684] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0685] Step 1:

[0686] The server receives a work document from the user. This received document is the data to be analyzed for natural language processing. The server converts this document into internal data and prepares it for use in the next analysis step.

[0687] Step 2:

[0688] The server analyzes the received work documents using a natural language processing toolkit (e.g., spaCy or NLTK). The data processing performed here involves extracting keywords and structural information from the documents. As a result of the analysis, the work procedures and important elements are extracted and organized into specific instructions.

[0689] Step 3:

[0690] The server selects the optimal machine control tool based on the analysis results. Specifically, it searches for libraries of corresponding automation tools (e.g., Ansible or Docker) based on the extracted keywords and determines the most suitable tool. The input is the analysis results, and the output is the selected control tool.

[0691] Step 4:

[0692] The server generates control codes using the selected machine control tool. This generation process executes templates and scripts for the selected tool to generate specific control codes. The input is the selected tool, and the output is the generated control code.

[0693] Step 5:

[0694] The server applies the generated control code to the target execution environment. Specifically, it sends the code to the PLC or SCADA system of the industrial facility and performs the processing necessary to start the automated operation of the equipment. The output is the result of the operation in the execution environment.

[0695] Step 6:

[0696] The server analyzes the execution results and reports them to the user. Specifically, it determines whether the operation was completed successfully or if there were any errors based on the data obtained from the execution environment. The input is the execution result data, and the output is the report content.

[0697] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0698] This invention provides a system that combines natural language processing capabilities with an emotion engine to enable analysis of work instructions, selection and execution of automated processes, and flexible responses based on the user's emotional reactions. This system operates server-centric, aiming to improve user operational efficiency and reduce mental burden.

[0699] The user sends a work order to the server via their terminal. The server uses natural language processing to analyze the content of the work order and identify the scope of automation. Based on the analysis results, it then selects the optimal automation tool and generates the necessary automation code. The generated code is applied in the configured execution environment, and the work is performed automatically.

[0700] In addition, the server includes an emotion engine that measures the user's emotional state while working and uses that information to further optimize the process. Specifically, it provides a more comfortable working environment by suggesting ways to simplify processes or increasing support when the user is feeling stressed. For example, when automating the email server setup process, the system enhances support by providing additional explanations and generating more detailed instructions if the user expresses anxiety.

[0701] In this way, the present invention can integrate analyzed data with emotional feedback to improve work efficiency and user experience. As a result, it achieves improved automation accuracy and quality of work procedures, reducing user errors and burdens.

[0702] The following describes the processing flow.

[0703] Step 1:

[0704] The user uploads the work order to the server using their terminal. The server starts operating in response to this action, receiving and saving the work order file.

[0705] Step 2:

[0706] The server activates its natural language processing function to analyze the contents of the work instructions. The analysis extracts important keywords and operational steps, and categorizes the procedures based on these.

[0707] Step 3:

[0708] Based on the analysis results, the server selects the most suitable automation tool. Depending on the nature and scope of the procedure, it selects tools such as those for system configuration changes or test automation.

[0709] Step 4:

[0710] The server generates automation code corresponding to the selected automation tool. The generated code includes specific operational instructions based on the work order and is formatted into an executable form.

[0711] Step 5:

[0712] The server applies the automated code generated in the execution environment. Before application, it verifies that it works correctly in the test environment, and if there are no problems, it is deployed to the production environment.

[0713] Step 6:

[0714] The server uses an emotion engine to monitor the user's emotional state while they are working. While the user interacts with the interface, it analyzes camera footage and input patterns to infer their feelings.

[0715] Step 7:

[0716] Based on the analysis results of the emotion engine, the server provides support tailored to the user. For example, if the user is feeling frustrated, it may present a tutorial or simplify an automated process.

[0717] Step 8:

[0718] After all processes are complete, the server generates a report of the execution results and provides it to the user. Furthermore, it updates the work procedure manual based on user feedback to improve future operations.

[0719] (Example 2)

[0720] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0721] Conventional automation technologies were limited to analyzing work instructions and generating and executing automation code, and were unable to respond flexibly while considering the user's emotional state. This could lead to user stress, making it difficult to improve work efficiency and usability. Furthermore, decisions regarding parts that were difficult to automate and updates to procedures based on execution results were often performed manually, resulting in a significant burden.

[0722] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0723] In this invention, the server includes means for analyzing the contents of an information instruction document, means for selecting an appropriate automation device based on the analyzed information instruction document, means for generating an automation program for the selected automation device, and means for evaluating the user's emotional state and making suggestions to optimize the work process based on that evaluation. This enables flexible automation that responds to the user's emotions, thereby improving work efficiency and usability.

[0724] "Natural language processing" refers to the technology used by computers to understand and analyze human language and extract its meaning.

[0725] An "information instruction sheet" is a document that contains instructions and details necessary for a task or process, written in natural language.

[0726] "Automated equipment" is a general term for hardware or software that automatically performs specified tasks according to a program.

[0727] An "automation program" is software code or a script designed to perform a specific task without human intervention.

[0728] The "implementation environment" refers to the environment, including the computer resources and system settings necessary for the automation program to run.

[0729] "Assessing a user's emotional state" is the process of detecting a user's emotional response and quantifying or categorizing that state.

[0730] "Suggestions for optimizing the work process" refer to improvement measures or alternative procedures instructed by the system to enhance the user's work efficiency and comfort.

[0731] This invention combines natural language processing capabilities with an emotion evaluation engine to provide analysis of work instructions, selection and execution of automated processes, and adaptive support based on user emotions. It operates server-centric, and the overall process is carried out as follows:

[0732] The user first sends a work instruction sheet to the server via their terminal. Since this instruction sheet is written in natural language, it can be used by users without special technical knowledge. The server receives the instruction sheet and analyzes it using natural language processing capabilities. For this analysis, a general-purpose machine learning library, for example, is used as the natural language processing library.

[0733] The server selects the appropriate automation device based on the analysis results. During the selection process, commonly used automation software, such as RPA solutions, may be utilized. Subsequently, an automation program is generated for the selected tool. This program is automatically created in script format and applied in the configured execution environment.

[0734] Furthermore, the server is equipped with technology for evaluating user emotions. This includes techniques such as facial recognition and voice analysis, and in some cases, a general-purpose emotion analysis API. If the server determines that a user is experiencing stress or anxiety during their work, it provides additional instructions and support. For example, it might display detailed guides in the user interface or provide supportive video instructions.

[0735] As a concrete example, suppose a user sends a work order to the server stating, "Automatically create the folder structure for a new project." In this case, the server analyzes the order, selects the appropriate folder generation program, and executes it in the configured environment. If the user expresses any concerns along the way, the server displays detailed instructions on the screen to help the user understand.

[0736] An example of a prompt message might be: "Please automate the setup process for new mail servers. Also, please provide detailed instructions for users who may feel unsure."

[0737] This embodiment of the invention makes it possible to realize an automated work process that is easy for users to operate and reduces their mental burden.

[0738] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0739] Step 1:

[0740] The user inputs a work instruction document written in natural language using a terminal and sends it to the server. This input is a document clearly describing the specific work to be done. The user's actions include creating and sending the instruction document. The server receives this instruction document and proceeds to the next analysis step.

[0741] Step 2:

[0742] The server analyzes received work orders using natural language processing capabilities. The input is the work order received from the user, and the output is the analyzed specific task information. Specific data processing includes text segmentation, keyword extraction, and contextual understanding. Based on the analysis results, the server determines which parts can be automated.

[0743] Step 3:

[0744] The server selects the most suitable automation device based on the analysis results. It receives the analyzed task information as input and the selected automation device as output. In this step, it searches the database for corresponding automation tools and scripts and selects the tool that can perform the task most efficiently.

[0745] Step 4:

[0746] The server generates automation programs for the selected automation devices. It takes the selection results as input and generates specific program code as output. Data calculations include script assembly and automatic generation of API calls.

[0747] Step 5:

[0748] The generated automation program is executed in the implementation environment. The server applies the program code to the implementation environment and automatically performs the configured tasks. The output is the result of the automated task execution, and the results are recorded in a predetermined format.

[0749] Step 6:

[0750] The server evaluates the user's emotional state during operation. Input is data such as the user's facial expressions and voice, and output is the evaluation result of the emotional state. Specific operations include real-time data capture and analysis.

[0751] Step 7:

[0752] Based on the user's emotional assessment, the server proposes optimizations to the work process. The emotional state assessment is taken as input, and the output is the optimized proposal. For example, if the server determines that the work is complex and stressful, it will offer suggestions to simplify the work procedure.

[0753] (Application Example 2)

[0754] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0755] In modern manufacturing, while there is a demand for automating processes based on work instructions, the challenge lies in reducing the mental burden and stress on workers and enabling them to perform tasks efficiently and flexibly. In particular, when workers feel anxious about complex instructions, there is a lack of appropriate procedure optimization and emotionally responsive support, and this problem needs to be addressed.

[0756] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0757] In this invention, the server includes means for analyzing the contents of work instructions, means for selecting an appropriate automation tool based on the analyzed work instructions, means for generating automation code for the selected automation tool, and means for acquiring emotional states and optimizing work procedures based on that information. This enables the selection and execution of automation processes from the analysis of work instructions, and further enables the provision of appropriate support according to the emotional state of the worker, thereby realizing an efficient and less stressful work environment.

[0758] "Natural language processing" refers to technologies that analyze input text data and understand its meaning and intent.

[0759] A "work instruction sheet" is a document that describes the procedures and conditions necessary to perform a specific task.

[0760] An "automation tool" is software or a system used to efficiently perform a specific task.

[0761] "Automation code" is a program written to execute a process automatically.

[0762] The "execution environment" refers to the hardware and software configuration required for a program to run.

[0763] "Emotional state" refers to a user's psychological or emotional condition, and measuring its changes is an indicator that can be used to support their work.

[0764] "Optimization" refers to improving existing processes and procedures to achieve a specific objective and maximize efficiency.

[0765] A system for implementing this invention consists of a server, a user terminal, and a work environment.

[0766] Program Overview

[0767] The server executes specific libraries (e.g., spaCy and Transformers) to perform natural language processing and analyzes work instructions received from the user's terminal. Based on the analysis results, it selects appropriate automation tools and generates program code. This automation code can then be run in an execution environment built on the server. Furthermore, the server uses an emotion analysis engine (e.g., IBM Watson) to evaluate the user's emotional state in real time and optimize the work procedure accordingly. For example, if the user is feeling anxious, it provides additional explanations or support.

[0768] Hardware and software used

[0769] Server: This is the central unit that analyzes work instructions and selects automation tools.

[0770] User terminal (e.g., smart glasses): A device used by workers to send work instructions to the server.

[0771] Emotion analysis engine: This is software used to measure a user's emotional state.

[0772] Specific example

[0773] Consider a scenario on a manufacturing line where a user puts on smart glasses and begins the setup procedure for a new machine. The user takes a picture of the work instructions through the glasses and sends them to a server. The server analyzes the information, generates the necessary automation code, and executes it. If the emotion engine detects the user's anxiety, it provides more specific setup instructions or visual guides to the glasses to aid understanding.

[0774] Example of a prompt

[0775] "Please explain the assembly procedure for the new parts. If there are any parts of the procedure that are complex or difficult to understand, please provide a detailed explanation."

[0776] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0777] Step 1:

[0778] The user uses smart glasses to photograph the work instruction sheet. At this time, the user captures the image of the work instruction sheet into their device and sends it to the server as digital data. The input is the image of the work instruction sheet, which forms the basis for subsequent data processing.

[0779] Step 2:

[0780] The server applies OCR technology to the received image of the work instruction and converts it into text data. This is data processing to convert image data into string data. The output is the text data of the work instruction.

[0781] Step 3:

[0782] The server uses a natural language processing library to analyze the text data of the work instructions in detail. This involves semantic and syntactic analysis of the text to identify tasks that can be automated. The input to this process is text data obtained by OCR, and the output is information about the analyzed tasks.

[0783] Step 4:

[0784] Based on the analyzed information, the server selects the optimal automation tool and generates the necessary automation code. The input is the task information obtained in step 3, and the output code is associated with a specific tool. At this stage, the server performs data calculations and creates code aimed at efficient task execution.

[0785] Step 5:

[0786] The generated automation code is run in a specific execution environment on the server. Here, the code is applied and its results are retrieved. The output of this process is the execution results and their report. The server monitors the execution process and collects results as needed.

[0787] Step 6:

[0788] The server uses an emotion analysis engine to monitor and evaluate the user's emotional state in real time. It analyzes the user's stress level and provides simplified procedures or additional explanations as needed. Inputs include the user's biometric data and facial expressions, and the analysis results are the output.

[0789] Step 7:

[0790] The user reviews the optimized instructions displayed on the smart glasses and proceeds with the task. Based on the sentiment analysis results obtained in Step 6, the server provides improvement suggestions, and the user utilizes specific work support. At this stage, the outputted instructions are presented in a way that is appropriate to the work environment.

[0791] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0792] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0793] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0794] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0795] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0796] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0797] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0798] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0799] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0800] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0801] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0802] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0803] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0804] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0805] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0806] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0807] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0808] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0809] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0810] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0811] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0812] The following is further disclosed regarding the embodiments described above.

[0813] (Claim 1)

[0814] A means for analyzing the contents of work instructions, which has natural language processing capabilities,

[0815] A means of selecting an appropriate automation tool based on the analyzed work instructions,

[0816] A means of generating automation code for the selected automation tool,

[0817] A means of applying the generated automation code to the execution environment and reporting the results,

[0818] A system that includes this.

[0819] (Claim 2)

[0820] The system according to claim 1, further comprising means for identifying tasks that are difficult to automate from the analyzed work instructions and presenting them to the person in charge.

[0821] (Claim 3)

[0822] The system according to claim 1, comprising means for automatically generating updated work procedure documents based on the execution results of the generated automation code.

[0823] "Example 1"

[0824] (Claim 1)

[0825] It has natural language processing capabilities and means for receiving work instructions,

[0826] A means of analyzing received work instructions to identify key procedures,

[0827] A means of selecting the optimal automation software based on the analysis results,

[0828] A means of generating automation scripts in a syntax suitable for the selected automation software,

[0829] A means of applying the generated script to the operational platform and recording and reporting the results,

[0830] A means of automatically generating documents after the completion of a task and providing reference materials to support subsequent tasks,

[0831] A system that includes this.

[0832] (Claim 2)

[0833] The system according to claim 1, comprising means for detecting processes requiring manual work from analyzed work instructions and notifying the person in charge.

[0834] (Claim 3)

[0835] The system according to claim 1, comprising means for automatically creating an updated work instruction document based on the execution results of a generated automation script.

[0836] "Application Example 1"

[0837] (Claim 1)

[0838] It has natural language processing capabilities and means for analyzing the content of work documents,

[0839] A means for selecting an appropriate machine control tool based on the analyzed work documents,

[0840] Means for generating control codes for selected machine control tools,

[0841] A means for applying the generated control code to the execution environment and reporting the results,

[0842] Means for automating the functions of industrial facilities,

[0843] A system that includes this.

[0844] (Claim 2)

[0845] The system according to claim 1, comprising means for identifying procedures that are difficult to automate from the analyzed work documents and presenting them to the person in charge.

[0846] (Claim 3)

[0847] The system according to claim 1, comprising means for automatically generating an updated work process document based on the execution result of the generated control code.

[0848] "Example 2 of combining an emotion engine"

[0849] (Claim 1)

[0850] A means for analyzing the contents of an information instruction document, which has natural language processing capabilities,

[0851] A means for selecting an appropriate automated device based on the analyzed information instruction sheet,

[0852] Means for generating an automation program for the selected automation equipment,

[0853] A means for applying the generated automation program to the implementation environment and reporting the results,

[0854] A means of evaluating the emotional state of users and proposing ways to optimize the work process based on that evaluation,

[0855] A system that includes this.

[0856] (Claim 2)

[0857] The system according to claim 1, further comprising means for identifying tasks that are difficult to automate from the analyzed information instruction sheets and presenting them to the person in charge.

[0858] (Claim 3)

[0859] The system according to claim 1, comprising means for automatically generating updated work procedure documents based on the results of the generated automation program.

[0860] "Application example 2 when combining with an emotional engine"

[0861] (Claim 1)

[0862] A means for analyzing the contents of work instructions, which has natural language processing capabilities,

[0863] A means of selecting an appropriate automation tool based on the analyzed work instructions,

[0864] A means of generating automation code for the selected automation tool,

[0865] A means of applying the generated automation code to the execution environment and reporting the results,

[0866] A means of acquiring emotional states and optimizing work procedures based on that information,

[0867] A system that includes this.

[0868] (Claim 2)

[0869] The system according to claim 1, further comprising means for identifying tasks that are difficult to automate from the analyzed work instructions and presenting them to the person in charge.

[0870] (Claim 3)

[0871] The system according to claim 1, comprising means for automatically generating updated work procedure documents based on the execution results of the generated automation code. [Explanation of Symbols]

[0872] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. It has natural language processing capabilities and means for analyzing the content of work documents, A means for selecting an appropriate machine control tool based on the analyzed work documents, Means for generating control codes for selected machine control tools, A means for applying the generated control code to the execution environment and reporting the results, Means for automating the functions of industrial facilities, A system that includes this.

2. The system according to claim 1, further comprising means for identifying procedures that are difficult to automate from the analyzed work documents and presenting them to the person in charge.

3. The system according to claim 1, comprising means for automatically generating an updated work process document based on the execution result of the generated control code.

Images