system

Abstract

We provide the system. [Solution] A means of observing the workflow and detecting repetitive tasks, A means of creating a process blueprint using a generative model, A means of converting design drawings into a format that can be imported into process automation tools, A means of monitoring automated processes and identifying areas for improvement, A system that includes means of providing an interactive agent to support communication.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern enterprises, the automation of business processes is essential for improving efficiency. However, with the current methods, it takes a great deal of time and effort to manually identify candidate processes for automation, create design drawings, and import them into tools. This has become a major obstacle to the rapid automation of business. In addition, there are problems that improving the automated process after introduction is laborious and lacks ease of communication.

Means for Solving the Problems

[0005] This invention provides a means for automatically observing business workflows and detecting repetitive tasks. Furthermore, it automatically creates process blueprints based on generative models and converts them into a format that can be directly imported into process automation tools. This automated process is monitored, and a mechanism for identifying improvement suggestions is in place. In addition, it provides an interactive agent function for smooth communication with the user and includes means for notifying users of improvement suggestions. This enables rapid and flexible business automation.

[0006] A "business process flow" is a diagram that shows the sequence of processes and work procedures within a company.

[0007] "Repetitive work" refers to a business process in which the same or similar procedures are performed repeatedly.

[0008] A "generative model" is an algorithm or mathematical method for automatically generating new outputs based on data.

[0009] A "process blueprint" is a document or format that provides detailed instructions and procedures for automating a specific business process.

[0010] "Process automation tools" are applications and platforms used to replace manual tasks performed by humans with computer software.

[0011] "Monitoring" is the act of observing whether a process or system is functioning correctly and collecting data to do so.

[0012] An "improvement proposal" refers to a modification or change suggested to make the current business processes or functions more efficient and effective.

[0013] A "conversational agent" is a software program that communicates with users using natural language to provide information and support problem-solving. [Brief explanation of the drawing]

[0014] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention provides an advanced system for automating corporate business processes, and specific embodiments thereof are shown below.

[0036] The system based on this invention primarily involves three parties: a server, a terminal, and a user. The user uses software on the terminal to perform their work activities. The terminal records the user's operation logs and data input history and transfers them to the server. The server aggregates the received data and analyzes common patterns in the workflow. This is achieved through algorithms utilizing machine learning.

[0037] The analysis automatically detects repetitive tasks and identifies processes that are candidates for RPA automation. These processes are then translated into natural language blueprints using generative models. These blueprints specify the process steps, required data, and outputs. These blueprints are then converted into a format understandable by the RPA tool and imported from the server to the tool. This automates the selected processes.

[0038] The execution of automated processes is continuously monitored by the server, and process performance data is collected and analyzed. This identifies areas that need improvement, and improvement suggestions are presented to the user. The user can review the provided improvement suggestions and provide feedback to the server as needed. This feedback helps to further improve the entire system.

[0039] Furthermore, the server incorporates an interactive agent, enabling real-time communication with users. This allows users to contact the server via their terminal to resolve questions or inquire about suggestions for process improvements.

[0040] A concrete example is a routine data entry task performed on a terminal at a certain company. The server observes this data entry work and identifies that the task is regular and repetitive. A blueprint is generated and imported into an automation tool, automating subsequent data entry tasks. As a result, users can concentrate their efforts on other important tasks.

[0041] Thus, by implementing the system of the present invention, it is possible to improve operational efficiency and reduce costs.

[0042] The following describes the processing flow.

[0043] Step 1:

[0044] The terminal observes the user's work operations in real time. It meticulously records operation logs and data entry timing, and prepares to send this information to the server in a format that can be used for later analysis.

[0045] Step 2:

[0046] The server collects operation log data sent from terminals and begins centralized management. Using machine learning algorithms, it automatically analyzes frequently occurring work patterns and procedures from the data. This extracts repetitive tasks and identifies processes that are candidates for RPA (Robotic Process Automation) implementation.

[0047] Step 3:

[0048] The server uses a generative model to create a blueprint in natural language, based on the details of the business processes identified through analysis. The blueprint clearly specifies the concrete steps, required datasets, and output specifications. This blueprint is then output in a format that RPA tools can accept.

[0049] Step 4:

[0050] The user uses a terminal to review the blueprints provided by the server and checks whether the process content matches their own workflow. They provide feedback on the blueprints, either approving them or requesting necessary modifications, and requesting corrections from the server as needed.

[0051] Step 5:

[0052] The server directly imports the user-approved blueprints into the RPA tool. This conversion is performed using scripted steps, and the automated process begins based on the imported blueprints.

[0053] Step 6:

[0054] The server continuously monitors automated processes in operation. It collects performance data for each process and monitors its operational status. Based on this, it identifies areas where improvement is possible and creates improvement plans based on the analysis results.

[0055] Step 7:

[0056] Users communicate questions and opinions to the server using the chatbot function on their devices. This interactive agent allows users to receive detailed explanations about improvement suggestions, helping them to deepen their understanding of business process automation.

[0057] (Example 1)

[0058] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0059] Traditional business processes often involve repetitive tasks, making efficient automation difficult. Furthermore, process improvement and optimization were limited due to insufficient communication with users. This invention aims to solve these problems and achieve business efficiency and optimization.

[0060] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0061] In this invention, the server includes means for observing business processes and detecting reproducible operations, means for creating a processing plan using a generation algorithm, and means for converting the plan into a format transferable to a processing automation means. This enables efficient automation of business processes and allows for the discovery and immediate response to areas for improvement through smooth communication with users.

[0062] "Business processes" refer to a series of business activities that companies and organizations perform on a daily basis, and these include repetitive tasks such as data entry and document creation.

[0063] "Reproducible operations" refer to routine user actions that are repeated many times under certain conditions, and which can usually be performed manually to achieve similar results.

[0064] A "generative algorithm" refers to a computational method or procedure that extracts specific patterns or features from data to create new information or blueprints.

[0065] A "plan" is a design document that details the procedures, requirements, and outputs for automating a business process.

[0066] "Process automation means" refers to methods and tools that use machines or software to execute business processes without human intervention.

[0067] "Interactive means" refers to means of mutual communication provided for the exchange of information and the receipt of instructions between a user and a system, and usually includes interfaces that use natural language.

[0068] "Information and communication means" refers to communication methods for transmitting data and information to users, and includes email and notification functions.

[0069] "Means of collecting user feedback and contributing to the overall improvement of the system" refers to methods of improving the system's performance and functionality by collecting and analyzing user feedback information.

[0070] The system of this invention is designed to efficiently automate and optimize a company's business processes. This system is primarily composed of a system in which a server, a terminal, and a user cooperate.

[0071] The server uses a high-performance computing system as the hardware necessary for information processing, and as software, it utilizes a "data processing framework" for data processing and "machine learning algorithms" for machine learning. Specifically, it integrates received business data and analyzes patterns based on the aggregated information.

[0072] A terminal is a device used by users to perform their daily tasks, and this includes typical computers and smart devices. Users input work data through the terminal, and this information is transmitted to the server.

[0073] Automated processes are continuously monitored by a server, and their performance data is collected. This collected data is then communicated to the user through the system's interactive mechanisms, providing suggestions for improvement. Users can then provide feedback based on these suggestions. This feedback contributes to further automation and overall system functionality improvements.

[0074] A concrete example is the data entry task that users perform on a daily basis. The server observes this operation, detects reproducible operations, and automatically creates a process plan using a generation algorithm. This plan is implemented in the system using process automation means, and the user's burden is significantly reduced by automating the work.

[0075] An example of a prompt using a generative AI model is, "Observe your daily data entry tasks and identify patterns that can be automated." This prompt allows the server to begin designing an effective automation process.

[0076] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0077] Step 1:

[0078] The terminal records user operation logs and data entry history and sends them to the server. The input includes all of the user's terminal operations. This data is sent to the server as a formatted log file. The server receives this file and stores it in its database.

[0079] Step 2:

[0080] The server aggregates the received operation logs using a "data processing framework." The input is log data sent from the terminal. The data is categorized, and unnecessary information is filtered out. This results in organized log data that provides a comprehensive overview of the business process.

[0081] Step 3:

[0082] The server uses a "machine learning algorithm" to analyze common patterns in business workflows based on aggregated data. Organized logs are used as input data, and patterns of specific business processes are obtained as output. In this step, different algorithms are tested during the analysis process, and the optimal model is selected.

[0083] Step 4:

[0084] The server detects processes that can be automated based on the analysis results and automatically generates process plans using a generative AI model. The input is the pattern analysis results, and the output is a plan that includes business procedures and necessary data. This specific operation is achieved through text generation utilizing natural language processing technology.

[0085] Step 5:

[0086] The server converts the generated plan into a "process automation tool" and formats it into a format that can be imported into the process automation tool. The input is a plan in natural language, and the output is a script or template that can be directly applied to the system. This step involves data transfer via an API.

[0087] Step 6:

[0088] The server monitors the execution of automated processes and continuously collects performance data. Inputs include log data generated during system execution. Outputs are metrics data indicating the efficiency and performance of the processes. This allows for real-time assessment of process health.

[0089] Step 7:

[0090] The server identifies areas for improvement based on the analyzed performance data and notifies the user. The input is the collected metrics data, and the output is recommended improvements. Specifically, it uses a notification system to send a message to the user's device.

[0091] Step 8:

[0092] The user reviews the improvement suggestions received from the server and provides feedback to the server as needed. The input is the received improvement suggestions, and the output is the user's evaluation and change requests. As a concrete example, opinions and suggestions can be sent using the feedback function on the device.

[0093] (Application Example 1)

[0094] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0095] In manufacturing environments, repetitive tasks performed by workers and their efficiency are crucial, but labor shortages and the complexity of the work can hinder efficient production. Furthermore, a lack of proper instructions on work procedures and suggestions for improvement often leads to decreased productivity. To address this, effective automation to support work and accurate, real-time feedback are necessary.

[0096] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0097] In this invention, the server includes means for observing work processes and identifying repetitive steps, means for creating process blueprints using generation algorithms, and means for converting the blueprints into a format that can be imported into process automation software. This enables efficient automation of repetitive steps and presentation of work procedures using a visual aid device.

[0098] "Observing the work process" means monitoring a series of tasks and procedures performed in manufacturing or service industries and evaluating the situation and progress.

[0099] "Recurring process discovery" is the process of identifying and extracting patterns in which similar tasks are performed regularly.

[0100] A "generative algorithm" is a program that uses machine learning or artificial intelligence technology to automatically derive patterns and rules from data.

[0101] "Creating a blueprint for a process" means visualizing or documenting specific work procedures or processes in a way that other systems and software can understand.

[0102] "Process automation software" is a program that automatically executes specific work procedures, minimizing the need for human intervention.

[0103] A "visual assistance device" is a device that provides visual information to human workers, helping them to perform their tasks efficiently and accurately.

[0104] A "generative AI model" is a framework for artificial intelligence that learns from large amounts of data to generate new information and predictions.

[0105] A "prompt sentence" is an input sentence used to give specific instructions or questions to an artificial intelligence model in order to obtain a result.

[0106] To implement this invention, it is necessary to build a system in which a server, a terminal, and a user work together. The server uses a machine learning-based generation algorithm to monitor the business process and identify repetitive steps. The terminal acts as an intermediate device that transmits user operation logs and data acquired from visual devices to the server.

[0107] The server primarily uses Python and Tensorflow® to analyze data and create process blueprints. These blueprints are then converted into a format that can be imported into RPA tools. This process automates identified repetitive steps.

[0108] Users can use smart glasses as a visual aid to visually receive work instructions, enabling them to proceed with the next task efficiently and accurately. This visual aid incorporates image processing libraries such as OpenCV, analyzing the work status in real time and providing necessary instructions to the user.

[0109] As a concrete example, the present invention can be applied to a screw-tightening process performed by workers on a manufacturing production line. In this case, the user receives real-time instructions such as "Tighten the next screw within 3 seconds" via smart glasses worn by the user. As a whole system, this improves work efficiency.

[0110] Possible prompts for the generated AI model include instructions such as, "Please provide information to improve the efficiency of the screw tightening process." Using these prompts, the generated AI model provides additional improvement suggestions and detailed procedural instructions to the user via the server.

[0111] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0112] Step 1:

[0113] The terminal collects user operation logs and data obtained from the visual device and sends them to the server. The input here is the user operation log and data from the visual device, which are integrated and converted into packets. Data packets are generated as output and forwarded to the server.

[0114] Step 2:

[0115] The server observes the business process based on the received data packets and identifies specific repetitive steps. It applies machine learning algorithms to extract patterns from the input data. This process outputs a list of repetitive steps.

[0116] Step 3:

[0117] Using a generative algorithm, the server creates a process blueprint. In this step, a machine learning model receives a list of iterative steps as input and generates a process blueprint based on it. The output is the process blueprint.

[0118] Step 4:

[0119] The server converts the generated blueprints into a format that can be imported into process automation software. The input is the blueprint, and the output is a file in a format that the automation tool can read. This conversion process enables subsequent automation.

[0120] Step 5:

[0121] The server's role is to monitor automated processes and collect performance data. It collects data on running processes as input and generates performance reports as output.

[0122] Step 6:

[0123] The user receives real-time work instructions through an interface provided via smart glasses. Input is instruction data received from a server, and output is visual instructions to the user. These instructions enable the worker to efficiently perform the next task.

[0124] Step 7:

[0125] The server automatically generates improvement suggestions and sends them as prompt messages to the AI ​​model, providing feedback to the user. The input is a performance report, and the output is a prompt message summarizing the improvement suggestions.

[0126] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0127] This invention provides a system for efficiently automating business processes and enabling operation that takes user emotions into consideration. The system features business flow observation, repetitive task detection, process automation (RPA), improvement suggestions, and the provision of an interactive agent. Furthermore, by incorporating an emotion engine, it enables interaction based on the user's emotional state.

[0128] This system primarily operates around servers, terminals, and users. First, when users perform their daily tasks, terminals record operation logs and transfer them to the server. The server uses this data to analyze common patterns in the workflow. Through this analysis, repetitive tasks are detected. The server then uses generative models to create detailed blueprints for these repetitive tasks and automates the business processes by directly importing them into an RPA tool.

[0129] After automating a business process, the server monitors the process and analyzes performance data. Based on the detected problems and potential improvements, it generates improvement suggestions and proposes them to the user. The user can review these suggestions on their terminal and provide feedback as needed.

[0130] Another feature of this invention is that an emotion engine is integrated into the server. This emotion engine detects the user's emotional state in real time during communication and adjusts the conversational agent's response based on that emotion. This provides users with a more user-friendly and effective interaction.

[0131] As a concrete example, consider call center operations. When an operator uses a terminal to input customer information and responds to inquiries, the server observes the operator's actions and identifies repetitive tasks. By introducing automated processes, operators are freed from repetitive input tasks and can concentrate on solving more complex problems. Furthermore, if the conversational agent detects the operator's stress through an emotion engine, it modifies its response to support the work. In this way, the system of the present invention provides an effective solution for simultaneously achieving efficiency and user comfort.

[0132] The following describes the processing flow.

[0133] Step 1:

[0134] The terminal monitors and records all operation logs and data inputs as the user performs their work activities. This information is prepared to be sent to the server in real time, as it forms the basis for subsequent analysis.

[0135] Step 2:

[0136] The server receives operation log data sent from terminals and centrally stores it in a database. Machine learning algorithms are used to analyze recurring patterns in the data and identify repetitive tasks.

[0137] Step 3:

[0138] The server generates a detailed blueprint of the business process, including repetitive tasks, based on the analysis results. This blueprint is created through a generative model and converted into a format acceptable to the RPA tool.

[0139] Step 4:

[0140] Users review the design drawings generated from the server via their terminals and evaluate whether their contents meet business requirements. If necessary, they provide feedback on the design drawings to the server, which then makes corrections.

[0141] Step 5:

[0142] The server imports the received blueprint into the RPA tool and starts the automation process. This process automates the identified repetitive tasks and executes them under the server's control.

[0143] Step 6:

[0144] The server monitors the running automation process in real time and analyzes the collected performance data. Based on this analysis, it determines the need for further improvements and generates optimization proposals.

[0145] Step 7:

[0146] When a user communicates with the server through an interactive agent, the emotion engine analyzes the user's emotional state. Based on this analysis, the server selects an appropriate communication style and provides responses that reduce the user's psychological burden.

[0147] (Example 2)

[0148] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0149] There is a need to simultaneously achieve efficiency improvements in business processes and enhance the quality of user interaction. While existing systems are automating business processes, they lack consideration for the emotional state of users, making it difficult to balance business efficiency and user satisfaction.

[0150] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0151] In this invention, the server includes means for monitoring business processes and identifying repetitive tasks, means for designing business processes using a generative artificial intelligence model, and means for recognizing emotional states and adjusting the responses of the dialogue system. This enables improved efficiency in business processes and effective information exchange that takes into account the user's emotions.

[0152] "Business processes" refer to a series of tasks and procedures performed within an organization or business, carried out to achieve a specific goal.

[0153] "Repetitive tasks" are those that involve repeating the same operations or procedures many times, and are usually expected to become more efficient through automation.

[0154] A "generative artificial intelligence model" is an AI technology that has the function of generating information using digital data and algorithms, and is used to optimize business processes.

[0155] A "business process" refers to the steps and flows necessary to carry out a business task, and is designed and adjusted for efficient execution.

[0156] A "task automation system" refers to a process that uses software or machines to streamline or eliminate manual work performed by humans.

[0157] "Information exchange" refers to the activity of transmitting and sharing information between people or systems, and is carried out for the purpose of mutual understanding and decision-making.

[0158] A "dialogue system" is software that enables communication between humans and computers, and generally uses speech recognition or natural language processing.

[0159] "Emotional state" refers to an individual's psychological condition and emotions, and is usually described using indicators that include stress levels and satisfaction levels.

[0160] This invention is a system that streamlines business processes and enables user-centric interactions, primarily involving servers, terminals, and users. Specific embodiments are described below.

[0161] The server collects business process information by monitoring the operations performed by users on their terminals. The terminals record a series of operation logs and transfer this data to the server. The server uses this data to analyze the business flow and identify repetitive tasks using specialized data analysis software, such as Python or R.

[0162] Based on the information from these iterative tasks, the server uses a generative artificial intelligence model to design the business process. This generative AI model may utilize open-source platforms such as TensorFlow or PyTorch. The generated design information is then imported into an automation system, such as UiPath or Automation Anywhere, to automate the business process.

[0163] Furthermore, the server incorporates sentiment analysis capabilities, collecting user interactions and analyzing the user's emotional state in real time. For example, if the server determines that the user is stressed, it adjusts its response through the dialogue system to provide a more user-friendly environment.

[0164] As a concrete example, in call center operations, when an operator inputs customer data on a terminal and responds to an inquiry, the server monitors the operation and automates repetitive tasks. As a result, operators can dedicate more time to handling more complex inquiries. In addition, an emotion engine can detect the operator's stress level and adjust the response accordingly, thereby improving the work environment.

[0165] An example of a prompt for a generated AI model is, "Detect a specific business procedure and create a detailed blueprint for automating it." This allows the server to provide information that helps streamline the given procedure.

[0166] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0167] Step 1:

[0168] When a user performs a task, the terminal records each operation in real time. Input includes user operation data (e.g., button clicks, input field data). Output is an operation log file, which details what and how the user performed the operation. This log file later serves as the basis for data analysis.

[0169] Step 2:

[0170] The terminal transfers operation logs recorded at regular intervals to the server. The input is an operation log file stored on the terminal. The output is business process log data stored in the server's database. This data is used for future data analysis on the server.

[0171] Step 3:

[0172] The server analyzes operation logs in the database to identify business processes. In particular, it utilizes a generative AI model to find patterns in repetitive tasks. The input is all operation logs stored in the database. The output is a list of identified business processes and repetitive tasks, which serves as the foundation for proceeding to the next automation step.

[0173] Step 4:

[0174] The server uses a generative AI model to create a blueprint for repetitive tasks. The input is the business flow data identified in step 3. The output is a detailed blueprint of the automatable business process, which is used to generate a template for input into the task automation system.

[0175] Step 5:

[0176] The server inputs the design blueprint into the automation system. In this case, an automation tool such as UiPath is used. The input is the created design blueprint template. The output is the program that executes the automated business process, which reduces manual work and improves efficiency.

[0177] Step 6:

[0178] The server continuously monitors automated business processes and collects performance data. Its input is real-time data obtained from running business processes. The output is a business performance report, which is used to identify problems and propose improvements.

[0179] Step 7:

[0180] The server uses an emotion engine to evaluate the user's emotional state during interactions and adjust the dialogue system's response accordingly. The input is user interaction data. The output is an adjusted response message, enabling dialogue that adapts to the user's emotions.

[0181] (Application Example 2)

[0182] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0183] In brick-and-mortar retail operations, there is a need to improve the efficiency of repetitive tasks performed by staff while providing flexible, emotion-based support. However, current systems are insufficient in automating processes that take staff emotional states into account and in adjusting support through conversational agents, which can lead to decreased efficiency and a decline in the quality of customer service.

[0184] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0185] In this invention, the server includes means for observing work activities and detecting repetitive tasks, means for creating process blueprints using generative models, and means for detecting the emotional state of workers and adjusting instructions and support content based on that state. This enables store staff to receive emotionally responsive support while improving operational efficiency through process automation.

[0186] "Business activities" refers to the collective work and processes that an organization or individual routinely undertakes to achieve a predetermined objective.

[0187] "Repetitive tasks" are those that involve performing the same procedure or task multiple times, and are usually tasks that can be automated.

[0188] A "generative model" is an algorithm or method for generating new information or structures based on data.

[0189] A "process blueprint" is a visual or descriptive representation of a business process that includes the details necessary for process automation.

[0190] "Process automation tools" are applications and platforms used to automate tasks performed by humans using machines or software.

[0191] "Monitoring" is the process of observing a series of activities or systems, and evaluating and recording their behavior and state.

[0192] An "improvement suggestion" is a proposal or opinion aimed at improving the efficiency and effectiveness of the current system or process.

[0193] A "conversational agent that facilitates communication" is a system or program designed to interact with users and promote smooth communication.

[0194] "Detecting an emotional state" means analyzing an individual's current emotions and identifying that state.

[0195] "Adjusting support content" means changing the type and method of support provided to an individual based on the information detected.

[0196] This application example is a system that enables improved operational efficiency and emotion-based staff support in physical stores. The server receives log data to record work activities and analyzes it to identify repetitive tasks. Based on the collected data, a generative model is used to create process blueprints, which are then imported into UiPath, a process automation tool. This automation process reduces the workload on staff.

[0197] Furthermore, the terminal uses an emotion detection algorithm based on TensorFlow to analyze the staff's emotional state in real time. Based on the analysis results, the server adjusts the instructions and support provided to the staff and notifies the store staff through an interactive agent that facilitates communication. This allows staff to receive appropriate support according to their emotional state.

[0198] For example, when store staff register product information several times a day, the terminal records this activity and adds it to an automated process on the server. Furthermore, if staff fatigue is detected, the terminal is notified with suggestions for breaks or, if possible, instructions to reassign the current task to another staff member.

[0199] An example of a prompt to input into the generating AI model is: "Manage repetitive tasks in an automated manner, taking into account the emotions of the staff. Suggest appropriate responses when staff are experiencing high levels of fatigue." This prompt is expected to enable the system to generate appropriate action plans, balancing operational efficiency with staff well-being.

[0200] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0201] Step 1:

[0202] The terminal records staff operation logs and sends them to the server. The input is data of the daily work operations performed by each staff member. The log data includes frequently performed operations, and this data is analyzed on the server side and used to identify repetitive tasks as common patterns. The output is operation log data expressed in a format suitable for analysis.

[0203] Step 2:

[0204] The server analyzes the received log data and identifies repetitive tasks. The input is log data sent from the terminal, and the output is a list of identified repetitive tasks. In this step, a generative AI model is used to analyze the data and extract tasks suitable for process automation.

[0205] Step 3:

[0206] The server leverages a generative model to create blueprints for repetitive tasks and converts them into a format that can be imported into process automation tools. The input is a list of identified repetitive tasks, and the output is blueprint data in a format suitable for process automation tools. This blueprint is then converted into a process that can be executed by subsequent process automation tools (e.g., UiPath).

[0207] Step 4:

[0208] The server uses a TensorFlow-based algorithm to receive and analyze staff emotional states from terminals in real time. The input is emotional data based on staff biometric information and actions, while the output is analyzed emotional state data. In this step, emotions are estimated using actual biometric data, and the results are used to support staff.

[0209] Step 5:

[0210] The server adjusts instructions and support for staff based on their emotional state and notifies them via their terminals. The input is analyzed emotional state data, and the output is emotionally appropriate instructions and support messages. For example, if fatigue is detected, a break suggestion will be displayed on the terminal.

[0211] Step 6:

[0212] Users receive support instructions and check the progress of tasks via their terminals. Inputs are instructions and support messages notified from the server, and outputs are the user's receipt of instructions and feedback. In this step, bidirectional communication between the server and the terminal ensures that tasks and support proceed smoothly.

[0213] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0214] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0215] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0216] [Second Embodiment]

[0217] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0218] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0219] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0220] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0221] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0222] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0223] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0224] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0225] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0226] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0227] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0228] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0229] This invention provides an advanced system for automating corporate business processes, and specific embodiments thereof are shown below.

[0230] The system based on this invention primarily involves three parties: a server, a terminal, and a user. The user uses software on the terminal to perform their work activities. The terminal records the user's operation logs and data input history and transfers them to the server. The server aggregates the received data and analyzes common patterns in the workflow. This is achieved through algorithms utilizing machine learning.

[0231] The analysis automatically detects repetitive tasks and identifies processes that are candidates for RPA automation. These processes are then translated into natural language blueprints using generative models. These blueprints specify the process steps, required data, and outputs. These blueprints are then converted into a format understandable by the RPA tool and imported from the server to the tool. This automates the selected processes.

[0232] The execution of automated processes is continuously monitored by the server, and process performance data is collected and analyzed. This identifies areas that need improvement, and improvement suggestions are presented to the user. The user can review the provided improvement suggestions and provide feedback to the server as needed. This feedback helps to further improve the entire system.

[0233] Furthermore, the server incorporates an interactive agent, enabling real-time communication with users. This allows users to contact the server via their terminal to resolve questions or inquire about suggestions for process improvements.

[0234] A concrete example is a routine data entry task performed on a terminal at a certain company. The server observes this data entry work and identifies that the task is regular and repetitive. A blueprint is generated and imported into an automation tool, automating subsequent data entry tasks. As a result, users can concentrate their efforts on other important tasks.

[0235] Thus, by implementing the system of the present invention, it is possible to improve operational efficiency and reduce costs.

[0236] The following describes the processing flow.

[0237] Step 1:

[0238] The terminal observes the user's work operations in real time. It meticulously records operation logs and data entry timing, and prepares to send this information to the server in a format that can be used for later analysis.

[0239] Step 2:

[0240] The server collects operation log data sent from terminals and begins centralized management. Using machine learning algorithms, it automatically analyzes frequently occurring work patterns and procedures from the data. This extracts repetitive tasks and identifies processes that are candidates for RPA (Robotic Process Automation) implementation.

[0241] Step 3:

[0242] The server uses a generative model to create a blueprint in natural language, based on the details of the business processes identified through analysis. The blueprint clearly specifies the concrete steps, required datasets, and output specifications. This blueprint is then output in a format that RPA tools can accept.

[0243] Step 4:

[0244] The user uses a terminal to review the blueprints provided by the server and checks whether the process content matches their own workflow. They provide feedback on the blueprints, either approving them or requesting necessary modifications, and requesting corrections from the server as needed.

[0245] Step 5:

[0246] The server directly imports the user-approved blueprints into the RPA tool. This conversion is performed using scripted steps, and the automated process begins based on the imported blueprints.

[0247] Step 6:

[0248] The server continuously monitors automated processes in operation. It collects performance data for each process and monitors its operational status. Based on this, it identifies areas where improvement is possible and creates improvement plans based on the analysis results.

[0249] Step 7:

[0250] Users communicate questions and opinions to the server using the chatbot function on their devices. This interactive agent allows users to receive detailed explanations about improvement suggestions, helping them to deepen their understanding of business process automation.

[0251] (Example 1)

[0252] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0253] Traditional business processes often involve repetitive tasks, making efficient automation difficult. Furthermore, process improvement and optimization were limited due to insufficient communication with users. This invention aims to solve these problems and achieve business efficiency and optimization.

[0254] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0255] In this invention, the server includes means for observing business processes and detecting reproducible operations, means for creating a processing plan using a generation algorithm, and means for converting the plan into a format transferable to a processing automation means. This enables efficient automation of business processes and allows for the discovery and immediate response to areas for improvement through smooth communication with users.

[0256] "Business processes" refer to a series of business activities that companies and organizations perform on a daily basis, and these include repetitive tasks such as data entry and document creation.

[0257] "Reproducible operations" refer to routine user actions that are repeated many times under certain conditions, and which can usually be performed manually to achieve similar results.

[0258] A "generative algorithm" refers to a computational method or procedure that extracts specific patterns or features from data to create new information or blueprints.

[0259] A "plan" is a design document that details the procedures, requirements, and outputs for automating a business process.

[0260] "Process automation means" refers to methods and tools that use machines or software to execute business processes without human intervention.

[0261] "Interactive means" refers to means of mutual communication provided for the exchange of information and the receipt of instructions between a user and a system, and usually includes interfaces that use natural language.

[0262] "Information and communication means" refers to communication methods for transmitting data and information to users, and includes email and notification functions.

[0263] "Means of collecting user feedback and contributing to the overall improvement of the system" refers to methods of improving the system's performance and functionality by collecting and analyzing user feedback information.

[0264] The system of this invention is designed to efficiently automate and optimize a company's business processes. This system is primarily composed of a system in which a server, a terminal, and a user cooperate.

[0265] The server uses a high-performance computing system as the hardware necessary for information processing, and as software, it utilizes a "data processing framework" for data processing and "machine learning algorithms" for machine learning. Specifically, it integrates received business data and analyzes patterns based on the aggregated information.

[0266] A terminal is a device used by users to perform their daily tasks, and this includes typical computers and smart devices. Users input work data through the terminal, and this information is transmitted to the server.

[0267] Automated processes are continuously monitored by a server, and their performance data is collected. This collected data is then communicated to the user through the system's interactive mechanisms, providing suggestions for improvement. Users can then provide feedback based on these suggestions. This feedback contributes to further automation and overall system functionality improvements.

[0268] A concrete example is the data entry task that users perform on a daily basis. The server observes this operation, detects reproducible operations, and automatically creates a process plan using a generation algorithm. This plan is implemented in the system using process automation means, and the user's burden is significantly reduced by automating the work.

[0269] An example of a prompt using a generative AI model is, "Observe your daily data entry tasks and identify patterns that can be automated." This prompt allows the server to begin designing an effective automation process.

[0270] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0271] Step 1:

[0272] The terminal records user operation logs and data entry history and sends them to the server. The input includes all of the user's terminal operations. This data is sent to the server as a formatted log file. The server receives this file and stores it in its database.

[0273] Step 2:

[0274] The server aggregates the received operation logs using a "data processing framework." The input is log data sent from the terminal. The data is categorized, and unnecessary information is filtered out. This results in organized log data that provides a comprehensive overview of the business process.

[0275] Step 3:

[0276] The server uses a "machine learning algorithm" to analyze common patterns in business workflows based on aggregated data. Organized logs are used as input data, and patterns of specific business processes are obtained as output. In this step, different algorithms are tested during the analysis process, and the optimal model is selected.

[0277] Step 4:

[0278] The server detects processes that can be automated based on the analysis results and automatically generates process plans using a generative AI model. The input is the pattern analysis results, and the output is a plan that includes business procedures and necessary data. This specific operation is achieved through text generation utilizing natural language processing technology.

[0279] Step 5:

[0280] The server converts the generated plan into a "process automation tool" and formats it into a format that can be imported into the process automation tool. The input is a plan in natural language, and the output is a script or template that can be directly applied to the system. This step involves data transfer via an API.

[0281] Step 6:

[0282] The server monitors the execution of the automated process and continuously collects performance data. The input includes the log data generated during system execution. The output is the metric data indicating the efficiency and performance of the process. This enables the real-time evaluation of the process's soundness.

[0283] Step 7:

[0284] Based on the analyzed performance data, the server identifies improvement points and notifies the user. The input is the collected metric data, and the output is the recommended improvement plan. As a specific operation, a notification system is used to send a message to the user's terminal.

[0285] Step 8:

[0286] The user reviews the improvement plan received from the server and returns feedback to the server if necessary. The input is the received improvement plan, and the output is the user's evaluation and change requests. As a specific example, the feedback function of the terminal is utilized to send opinions and suggestions.

[0287] (Application Example 1)

[0288] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0289] In a manufacturing site, repetitive tasks performed by workers and their efficiency improvement are important, but efficient production may be hindered due to labor shortages and work complexity. There is also a problem that productivity is likely to decrease due to insufficient appropriate instructions and improvement plan presentations for work procedures. To solve this, effective work support by automation and accurate real-time feedback are required.

[0290] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0291] In this invention, the server includes means for observing work processes and identifying repetitive steps, means for creating process blueprints using generation algorithms, and means for converting the blueprints into a format that can be imported into process automation software. This enables efficient automation of repetitive steps and presentation of work procedures using a visual aid device.

[0292] "Observing the work process" means monitoring a series of tasks and procedures performed in manufacturing or service industries and evaluating the situation and progress.

[0293] "Recurring process discovery" is the process of identifying and extracting patterns in which similar tasks are performed regularly.

[0294] A "generative algorithm" is a program that uses machine learning or artificial intelligence technology to automatically derive patterns and rules from data.

[0295] "Creating a blueprint for a process" means visualizing or documenting specific work procedures or processes in a way that other systems and software can understand.

[0296] "Process automation software" is a program that automatically executes specific work procedures, minimizing the need for human intervention.

[0297] A "visual assistance device" is a device that provides visual information to human workers, helping them to perform their tasks efficiently and accurately.

[0298] A "generative AI model" is a framework for artificial intelligence that learns from large amounts of data to generate new information and predictions.

[0299] A "prompt sentence" is an input sentence used to give specific instructions or questions to an artificial intelligence model in order to obtain a result.

[0300] To implement this invention, it is necessary to build a system in which a server, a terminal, and a user work together. The server uses a machine learning-based generation algorithm to monitor the business process and identify repetitive steps. The terminal acts as an intermediate device that transmits user operation logs and data acquired from visual devices to the server.

[0301] The server primarily uses Python and TensorFlow to analyze data and create process blueprints. These blueprints are then converted into a format that can be imported into RPA tools. This process automates identified repetitive steps.

[0302] Users can use smart glasses as a visual aid to visually receive work instructions, enabling them to proceed with the next task efficiently and accurately. This visual aid incorporates image processing libraries such as OpenCV, analyzing the work status in real time and providing necessary instructions to the user.

[0303] As a concrete example, the present invention can be applied to a screw-tightening process performed by workers on a manufacturing production line. In this case, the user receives real-time instructions such as "Tighten the next screw within 3 seconds" via smart glasses worn by the user. As a whole system, this improves work efficiency.

[0304] Possible prompts for the generated AI model include instructions such as, "Please provide information to improve the efficiency of the screw tightening process." Using these prompts, the generated AI model provides additional improvement suggestions and detailed procedural instructions to the user via the server.

[0305] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0306] Step 1:

[0307] The terminal collects data obtained from the user's operation logs and visual devices and sends it to the server. The input here is the user's operation logs and visual device data, which are integrated and converted into packets. A data packet to be transferred to the server is generated as the output.

[0308] Step 2:

[0309] Based on the received data packet, the server observes the business process and discovers specific repetitive processes. It applies a machine learning algorithm to extract patterns from the input data. As a result of this operation, a list of repetitive processes is output.

[0310] Step 3:

[0311] Using a generation algorithm, the server creates a design drawing of the process. In this step, the machine learning model receives the list of repetitive processes as input and generates a design drawing of the process based on it. The output is the design drawing of the process.

[0312] Step 4:

[0313] The server converts the generated design drawing into a format that can be imported into process automation software. The input is the design drawing, and the output is a file in a format that can be read by an automation tool. This conversion process enables subsequent automation.

[0314] Step 5:

[0315] It is the role of the server to monitor the automated process and collect performance data. It collects the running process data as input and generates a performance report as the output.

[0316] Step 6:

[0317] The user receives real-time work instructions through an interface provided via smart glasses. Input is instruction data received from a server, and output is visual instructions to the user. These instructions enable the worker to efficiently perform the next task.

[0318] Step 7:

[0319] The server automatically generates improvement suggestions and sends them as prompt messages to the AI ​​model, providing feedback to the user. The input is a performance report, and the output is a prompt message summarizing the improvement suggestions.

[0320] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0321] This invention provides a system for efficiently automating business processes and enabling operation that takes user emotions into consideration. The system features business flow observation, repetitive task detection, process automation (RPA), improvement suggestions, and the provision of an interactive agent. Furthermore, by incorporating an emotion engine, it enables interaction based on the user's emotional state.

[0322] This system primarily operates around servers, terminals, and users. First, when users perform their daily tasks, terminals record operation logs and transfer them to the server. The server uses this data to analyze common patterns in the workflow. Through this analysis, repetitive tasks are detected. The server then uses generative models to create detailed blueprints for these repetitive tasks and automates the business processes by directly importing them into an RPA tool.

[0323] After automating a business process, the server monitors the process and analyzes performance data. Based on the detected problems and potential improvements, it generates improvement suggestions and proposes them to the user. The user can review these suggestions on their terminal and provide feedback as needed.

[0324] Another feature of this invention is that an emotion engine is integrated into the server. This emotion engine detects the user's emotional state in real time during communication and adjusts the conversational agent's response based on that emotion. This provides users with a more user-friendly and effective interaction.

[0325] As a concrete example, consider call center operations. When an operator uses a terminal to input customer information and responds to inquiries, the server observes the operator's actions and identifies repetitive tasks. By introducing automated processes, operators are freed from repetitive input tasks and can concentrate on solving more complex problems. Furthermore, if the conversational agent detects the operator's stress through an emotion engine, it modifies its response to support the work. In this way, the system of the present invention provides an effective solution for simultaneously achieving efficiency and user comfort.

[0326] The following describes the processing flow.

[0327] Step 1:

[0328] The terminal monitors and records all operation logs and data inputs as the user performs their work activities. This information is prepared to be sent to the server in real time, as it forms the basis for subsequent analysis.

[0329] Step 2:

[0330] The server receives operation log data sent from terminals and centrally stores it in a database. Machine learning algorithms are used to analyze recurring patterns in the data and identify repetitive tasks.

[0331] Step 3:

[0332] The server generates a detailed blueprint of the business process, including repetitive tasks, based on the analysis results. This blueprint is created through a generative model and converted into a format acceptable to the RPA tool.

[0333] Step 4:

[0334] Users review the design drawings generated from the server via their terminals and evaluate whether their contents meet business requirements. If necessary, they provide feedback on the design drawings to the server, which then makes corrections.

[0335] Step 5:

[0336] The server imports the received blueprint into the RPA tool and starts the automation process. This process automates the identified repetitive tasks and executes them under the server's control.

[0337] Step 6:

[0338] The server monitors the running automation process in real time and analyzes the collected performance data. Based on this analysis, it determines the need for further improvements and generates optimization proposals.

[0339] Step 7:

[0340] When a user communicates with the server through an interactive agent, the emotion engine analyzes the user's emotional state. Based on this analysis, the server selects an appropriate communication style and provides responses that reduce the user's psychological burden.

[0341] (Example 2)

[0342] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0343] There is a need to simultaneously achieve efficiency improvements in business processes and enhance the quality of user interaction. While existing systems are automating business processes, they lack consideration for the emotional state of users, making it difficult to balance business efficiency and user satisfaction.

[0344] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0345] In this invention, the server includes means for monitoring business processes and identifying repetitive tasks, means for designing business processes using a generative artificial intelligence model, and means for recognizing emotional states and adjusting the responses of the dialogue system. This enables improved efficiency in business processes and effective information exchange that takes into account the user's emotions.

[0346] "Business processes" refer to a series of tasks and procedures performed within an organization or business, carried out to achieve a specific goal.

[0347] "Repetitive tasks" are those that involve repeating the same operations or procedures many times, and are usually expected to become more efficient through automation.

[0348] A "generative artificial intelligence model" is an AI technology that has the function of generating information using digital data and algorithms, and is used to optimize business processes.

[0349] A "business process" refers to the steps and flows necessary to carry out a business task, and is designed and adjusted for efficient execution.

[0350] A "task automation system" refers to a process that uses software or machines to streamline or eliminate manual work performed by humans.

[0351] "Information exchange" refers to the activity of transmitting and sharing information between people or systems, and is carried out for the purpose of mutual understanding and decision-making.

[0352] A "dialogue system" is software that enables communication between humans and computers, and generally uses speech recognition or natural language processing.

[0353] "Emotional state" refers to an individual's psychological condition and emotions, and is usually described using indicators that include stress levels and satisfaction levels.

[0354] This invention is a system that streamlines business processes and enables user-centric interactions, primarily involving servers, terminals, and users. Specific embodiments are described below.

[0355] The server collects business process information by monitoring the operations performed by users on their terminals. The terminals record a series of operation logs and transfer this data to the server. The server uses this data to analyze the business flow and identify repetitive tasks using specialized data analysis software, such as Python or R.

[0356] Based on the information from these iterative tasks, the server uses a generative artificial intelligence model to design the business process. This generative AI model may utilize open-source platforms such as TensorFlow or PyTorch. The generated design information is then imported into an automation system, such as UiPath or Automation Anywhere, to automate the business process.

[0357] Furthermore, the server incorporates sentiment analysis capabilities, collecting user interactions and analyzing the user's emotional state in real time. For example, if the server determines that the user is stressed, it adjusts its response through the dialogue system to provide a more user-friendly environment.

[0358] As a concrete example, in call center operations, when an operator inputs customer data on a terminal and responds to an inquiry, the server monitors the operation and automates repetitive tasks. As a result, operators can dedicate more time to handling more complex inquiries. In addition, an emotion engine can detect the operator's stress level and adjust the response accordingly, thereby improving the work environment.

[0359] An example of a prompt for a generated AI model is, "Detect a specific business procedure and create a detailed blueprint for automating it." This allows the server to provide information that helps streamline the given procedure.

[0360] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0361] Step 1:

[0362] When a user performs a task, the terminal records each operation in real time. Input includes user operation data (e.g., button clicks, input field data). Output is an operation log file, which details what and how the user performed the operation. This log file later serves as the basis for data analysis.

[0363] Step 2:

[0364] The terminal transfers operation logs recorded at regular intervals to the server. The input is an operation log file stored on the terminal. The output is business process log data stored in the server's database. This data is used for future data analysis on the server.

[0365] Step 3:

[0366] The server analyzes operation logs in the database to identify business processes. In particular, it utilizes a generative AI model to find patterns in repetitive tasks. The input is all operation logs stored in the database. The output is a list of identified business processes and repetitive tasks, which serves as the foundation for proceeding to the next automation step.

[0367] Step 4:

[0368] The server uses a generative AI model to create a blueprint for repetitive tasks. The input is the business flow data identified in step 3. The output is a detailed blueprint of the automatable business process, which is used to generate a template for input into the task automation system.

[0369] Step 5:

[0370] The server inputs the design blueprint into the automation system. In this case, an automation tool such as UiPath is used. The input is the created design blueprint template. The output is the program that executes the automated business process, which reduces manual work and improves efficiency.

[0371] Step 6:

[0372] The server continuously monitors automated business processes and collects performance data. Its input is real-time data obtained from running business processes. The output is a business performance report, which is used to identify problems and propose improvements.

[0373] Step 7:

[0374] The server uses an emotion engine to evaluate the user's emotional state during interactions and adjust the dialogue system's response accordingly. The input is user interaction data. The output is an adjusted response message, enabling dialogue that adapts to the user's emotions.

[0375] (Application Example 2)

[0376] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0377] In brick-and-mortar retail operations, there is a need to improve the efficiency of repetitive tasks performed by staff while providing flexible, emotion-based support. However, current systems are insufficient in automating processes that take staff emotional states into account and in adjusting support through conversational agents, which can lead to decreased efficiency and a decline in the quality of customer service.

[0378] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0379] In this invention, the server includes means for observing work activities and detecting repetitive tasks, means for creating process blueprints using generative models, and means for detecting the emotional state of workers and adjusting instructions and support content based on that state. This enables store staff to receive emotionally responsive support while improving operational efficiency through process automation.

[0380] "Business activities" refers to the collective work and processes that an organization or individual routinely undertakes to achieve a predetermined objective.

[0381] "Repetitive tasks" are those that involve performing the same procedure or task multiple times, and are usually tasks that can be automated.

[0382] A "generative model" is an algorithm or method for generating new information or structures based on data.

[0383] A "process blueprint" is a visual or descriptive representation of a business process that includes the details necessary for process automation.

[0384] "Process automation tools" are applications and platforms used to automate tasks performed by humans using machines or software.

[0385] "Monitoring" is the process of observing a series of activities or systems, and evaluating and recording their behavior and state.

[0386] An "improvement suggestion" is a proposal or opinion aimed at improving the efficiency and effectiveness of the current system or process.

[0387] A "conversational agent that facilitates communication" is a system or program designed to interact with users and promote smooth communication.

[0388] "Detecting an emotional state" means analyzing an individual's current emotions and identifying that state.

[0389] "Adjusting support content" means changing the type and method of support provided to an individual based on the information detected.

[0390] This application example is a system that enables improved operational efficiency and emotion-based staff support in physical stores. The server receives log data to record work activities and analyzes it to identify repetitive tasks. Based on the collected data, a generative model is used to create process blueprints, which are then imported into UiPath, a process automation tool. This automation process reduces the workload on staff.

[0391] Furthermore, the terminal uses an emotion detection algorithm based on TensorFlow to analyze the staff's emotional state in real time. Based on the analysis results, the server adjusts the instructions and support provided to the staff and notifies the store staff through an interactive agent that facilitates communication. This allows staff to receive appropriate support according to their emotional state.

[0392] For example, when store staff register product information several times a day, the terminal records this activity and adds it to an automated process on the server. Furthermore, if staff fatigue is detected, the terminal is notified with suggestions for breaks or, if possible, instructions to reassign the current task to another staff member.

[0393] An example of a prompt to input into the generating AI model is: "Manage repetitive tasks in an automated manner, taking into account the emotions of the staff. Suggest appropriate responses when staff are experiencing high levels of fatigue." This prompt is expected to enable the system to generate appropriate action plans, balancing operational efficiency with staff well-being.

[0394] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0395] Step 1:

[0396] The terminal records staff operation logs and sends them to the server. The input is data of the daily work operations performed by each staff member. The log data includes frequently performed operations, and this data is analyzed on the server side and used to identify repetitive tasks as common patterns. The output is operation log data expressed in a format suitable for analysis.

[0397] Step 2:

[0398] The server analyzes the received log data and identifies repetitive tasks. The input is log data sent from the terminal, and the output is a list of identified repetitive tasks. In this step, a generative AI model is used to analyze the data and extract tasks suitable for process automation.

[0399] Step 3:

[0400] The server leverages a generative model to create blueprints for repetitive tasks and converts them into a format that can be imported into process automation tools. The input is a list of identified repetitive tasks, and the output is blueprint data in a format suitable for process automation tools. This blueprint is then converted into a process that can be executed by subsequent process automation tools (e.g., UiPath).

[0401] Step 4:

[0402] The server uses a TensorFlow-based algorithm to receive and analyze staff emotional states from terminals in real time. The input is emotional data based on staff biometric information and actions, while the output is analyzed emotional state data. In this step, emotions are estimated using actual biometric data, and the results are used to support staff.

[0403] Step 5:

[0404] The server adjusts instructions and support for staff based on their emotional state and notifies them via their terminals. The input is analyzed emotional state data, and the output is emotionally appropriate instructions and support messages. For example, if fatigue is detected, a break suggestion will be displayed on the terminal.

[0405] Step 6:

[0406] Users receive support instructions and check the progress of tasks via their terminals. Inputs are instructions and support messages notified from the server, and outputs are the user's receipt of instructions and feedback. In this step, bidirectional communication between the server and the terminal ensures that tasks and support proceed smoothly.

[0407] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0408] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0409] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0410] [Third Embodiment]

[0411] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0412] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0413] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0414] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0415] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0416] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0417] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0418] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0419] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0420] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0421] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0422] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0423] This invention provides an advanced system for automating corporate business processes, and specific embodiments thereof are shown below.

[0424] The system based on this invention primarily involves three parties: a server, a terminal, and a user. The user uses software on the terminal to perform their work activities. The terminal records the user's operation logs and data input history and transfers them to the server. The server aggregates the received data and analyzes common patterns in the workflow. This is achieved through algorithms utilizing machine learning.

[0425] The analysis automatically detects repetitive tasks and identifies processes that are candidates for RPA automation. These processes are then translated into natural language blueprints using generative models. These blueprints specify the process steps, required data, and outputs. These blueprints are then converted into a format understandable by the RPA tool and imported from the server to the tool. This automates the selected processes.

[0426] The execution of automated processes is continuously monitored by the server, and process performance data is collected and analyzed. This identifies areas that need improvement, and improvement suggestions are presented to the user. The user can review the provided improvement suggestions and provide feedback to the server as needed. This feedback helps to further improve the entire system.

[0427] Furthermore, the server incorporates an interactive agent, enabling real-time communication with users. This allows users to contact the server via their terminal to resolve questions or inquire about suggestions for process improvements.

[0428] A concrete example is a routine data entry task performed on a terminal at a certain company. The server observes this data entry work and identifies that the task is regular and repetitive. A blueprint is generated and imported into an automation tool, automating subsequent data entry tasks. As a result, users can concentrate their efforts on other important tasks.

[0429] Thus, by implementing the system of the present invention, it is possible to improve operational efficiency and reduce costs.

[0430] The following describes the processing flow.

[0431] Step 1:

[0432] The terminal observes the user's work operations in real time. It meticulously records operation logs and data entry timing, and prepares to send this information to the server in a format that can be used for later analysis.

[0433] Step 2:

[0434] The server collects operation log data sent from terminals and begins centralized management. Using machine learning algorithms, it automatically analyzes frequently occurring work patterns and procedures from the data. This extracts repetitive tasks and identifies processes that are candidates for RPA (Robotic Process Automation) implementation.

[0435] Step 3:

[0436] The server uses a generative model to create a blueprint in natural language, based on the details of the business processes identified through analysis. The blueprint clearly specifies the concrete steps, required datasets, and output specifications. This blueprint is then output in a format that RPA tools can accept.

[0437] Step 4:

[0438] The user uses a terminal to review the blueprints provided by the server and checks whether the process content matches their own workflow. They provide feedback on the blueprints, either approving them or requesting necessary modifications, and requesting corrections from the server as needed.

[0439] Step 5:

[0440] The server directly imports the user-approved blueprints into the RPA tool. This conversion is performed using scripted steps, and the automated process begins based on the imported blueprints.

[0441] Step 6:

[0442] The server continuously monitors automated processes in operation. It collects performance data for each process and monitors its operational status. Based on this, it identifies areas where improvement is possible and creates improvement plans based on the analysis results.

[0443] Step 7:

[0444] Users communicate questions and opinions to the server using the chatbot function on their devices. This interactive agent allows users to receive detailed explanations about improvement suggestions, helping them to deepen their understanding of business process automation.

[0445] (Example 1)

[0446] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0447] Traditional business processes often involve repetitive tasks, making efficient automation difficult. Furthermore, process improvement and optimization were limited due to insufficient communication with users. This invention aims to solve these problems and achieve business efficiency and optimization.

[0448] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0449] In this invention, the server includes means for observing business processes and detecting reproducible operations, means for creating a processing plan using a generation algorithm, and means for converting the plan into a format transferable to a processing automation means. This enables efficient automation of business processes and allows for the discovery and immediate response to areas for improvement through smooth communication with users.

[0450] "Business processes" refer to a series of business activities that companies and organizations perform on a daily basis, and these include repetitive tasks such as data entry and document creation.

[0451] "Reproducible operations" refer to routine user actions that are repeated many times under certain conditions, and which can usually be performed manually to achieve similar results.

[0452] A "generative algorithm" refers to a computational method or procedure that extracts specific patterns or features from data to create new information or blueprints.

[0453] A "plan" is a design document that details the procedures, requirements, and outputs for automating a business process.

[0454] "Process automation means" refers to methods and tools that use machines or software to execute business processes without human intervention.

[0455] "Interactive means" refers to means of mutual communication provided for the exchange of information and the receipt of instructions between a user and a system, and usually includes interfaces that use natural language.

[0456] "Information and communication means" refers to communication methods for transmitting data and information to users, and includes email and notification functions.

[0457] "Means of collecting user feedback and contributing to the overall improvement of the system" refers to methods of improving the system's performance and functionality by collecting and analyzing user feedback information.

[0458] The system of this invention is designed to efficiently automate and optimize a company's business processes. This system is primarily composed of a system in which a server, a terminal, and a user cooperate.

[0459] The server uses a high-performance computing system as the hardware necessary for information processing, and as software, it utilizes a "data processing framework" for data processing and "machine learning algorithms" for machine learning. Specifically, it integrates received business data and analyzes patterns based on the aggregated information.

[0460] A terminal is a device used by users to perform their daily tasks, and this includes typical computers and smart devices. Users input work data through the terminal, and this information is transmitted to the server.

[0461] Automated processes are continuously monitored by a server, and their performance data is collected. This collected data is then communicated to the user through the system's interactive mechanisms, providing suggestions for improvement. Users can then provide feedback based on these suggestions. This feedback contributes to further automation and overall system functionality improvements.

[0462] A concrete example is the data entry task that users perform on a daily basis. The server observes this operation, detects reproducible operations, and automatically creates a process plan using a generation algorithm. This plan is implemented in the system using process automation means, and the user's burden is significantly reduced by automating the work.

[0463] An example of a prompt using a generative AI model is, "Observe your daily data entry tasks and identify patterns that can be automated." This prompt allows the server to begin designing an effective automation process.

[0464] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0465] Step 1:

[0466] The terminal records user operation logs and data entry history and sends them to the server. The input includes all of the user's terminal operations. This data is sent to the server as a formatted log file. The server receives this file and stores it in its database.

[0467] Step 2:

[0468] The server aggregates the received operation logs using a "data processing framework." The input is log data sent from the terminal. The data is categorized, and unnecessary information is filtered out. This results in organized log data that provides a comprehensive overview of the business process.

[0469] Step 3:

[0470] The server uses a "machine learning algorithm" to analyze common patterns in business workflows based on aggregated data. Organized logs are used as input data, and patterns of specific business processes are obtained as output. In this step, different algorithms are tested during the analysis process, and the optimal model is selected.

[0471] Step 4:

[0472] The server detects processes that can be automated based on the analysis results and automatically generates process plans using a generative AI model. The input is the pattern analysis results, and the output is a plan that includes business procedures and necessary data. This specific operation is achieved through text generation utilizing natural language processing technology.

[0473] Step 5:

[0474] The server converts the generated plan into a "process automation tool" and formats it into a format that can be imported into the process automation tool. The input is a plan in natural language, and the output is a script or template that can be directly applied to the system. This step involves data transfer via an API.

[0475] Step 6:

[0476] The server monitors the execution of automated processes and continuously collects performance data. Inputs include log data generated during system execution. Outputs are metrics data indicating the efficiency and performance of the processes. This allows for real-time assessment of process health.

[0477] Step 7:

[0478] The server identifies areas for improvement based on the analyzed performance data and notifies the user. The input is the collected metrics data, and the output is recommended improvements. Specifically, it uses a notification system to send a message to the user's device.

[0479] Step 8:

[0480] The user reviews the improvement suggestions received from the server and provides feedback to the server as needed. The input is the received improvement suggestions, and the output is the user's evaluation and change requests. As a concrete example, opinions and suggestions can be sent using the feedback function on the device.

[0481] (Application Example 1)

[0482] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0483] In manufacturing environments, repetitive tasks performed by workers and their efficiency are crucial, but labor shortages and the complexity of the work can hinder efficient production. Furthermore, a lack of proper instructions on work procedures and suggestions for improvement often leads to decreased productivity. To address this, effective automation to support work and accurate, real-time feedback are necessary.

[0484] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0485] In this invention, the server includes means for observing work processes and identifying repetitive steps, means for creating process blueprints using generation algorithms, and means for converting the blueprints into a format that can be imported into process automation software. This enables efficient automation of repetitive steps and presentation of work procedures using a visual aid device.

[0486] "Observing the work process" means monitoring a series of tasks and procedures performed in manufacturing or service industries and evaluating the situation and progress.

[0487] "Recurring process discovery" is the process of identifying and extracting patterns in which similar tasks are performed regularly.

[0488] A "generative algorithm" is a program that uses machine learning or artificial intelligence technology to automatically derive patterns and rules from data.

[0489] "Creating a blueprint for a process" means visualizing or documenting specific work procedures or processes in a way that other systems and software can understand.

[0490] "Process automation software" is a program that automatically executes specific work procedures, minimizing the need for human intervention.

[0491] A "visual assistance device" is a device that provides visual information to human workers, helping them to perform their tasks efficiently and accurately.

[0492] A "generative AI model" is a framework for artificial intelligence that learns from large amounts of data to generate new information and predictions.

[0493] A "prompt sentence" is an input sentence used to give specific instructions or questions to an artificial intelligence model in order to obtain a result.

[0494] To implement this invention, it is necessary to build a system in which a server, a terminal, and a user work together. The server uses a machine learning-based generation algorithm to monitor the business process and identify repetitive steps. The terminal acts as an intermediate device that transmits user operation logs and data acquired from visual devices to the server.

[0495] The server primarily uses Python and TensorFlow to analyze data and create process blueprints. These blueprints are then converted into a format that can be imported into RPA tools. This process automates identified repetitive steps.

[0496] Users can use smart glasses as a visual aid to visually receive work instructions, enabling them to proceed with the next task efficiently and accurately. This visual aid incorporates image processing libraries such as OpenCV, analyzing the work status in real time and providing necessary instructions to the user.

[0497] As a concrete example, the present invention can be applied to a screw-tightening process performed by workers on a manufacturing production line. In this case, the user receives real-time instructions such as "Tighten the next screw within 3 seconds" via smart glasses worn by the user. As a whole system, this improves work efficiency.

[0498] Possible prompts for the generated AI model include instructions such as, "Please provide information to improve the efficiency of the screw tightening process." Using these prompts, the generated AI model provides additional improvement suggestions and detailed procedural instructions to the user via the server.

[0499] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0500] Step 1:

[0501] The terminal collects user operation logs and data obtained from the visual device and sends them to the server. The input here is the user operation log and data from the visual device, which are integrated and converted into packets. Data packets are generated as output and forwarded to the server.

[0502] Step 2:

[0503] The server observes the business process based on the received data packets and identifies specific repetitive steps. It applies machine learning algorithms to extract patterns from the input data. This process outputs a list of repetitive steps.

[0504] Step 3:

[0505] Using a generative algorithm, the server creates a process blueprint. In this step, a machine learning model receives a list of iterative steps as input and generates a process blueprint based on it. The output is the process blueprint.

[0506] Step 4:

[0507] The server converts the generated blueprints into a format that can be imported into process automation software. The input is the blueprint, and the output is a file in a format that the automation tool can read. This conversion process enables subsequent automation.

[0508] Step 5:

[0509] The server's role is to monitor automated processes and collect performance data. It collects data on running processes as input and generates performance reports as output.

[0510] Step 6:

[0511] The user receives real-time work instructions through an interface provided via smart glasses. Input is instruction data received from a server, and output is visual instructions to the user. These instructions enable the worker to efficiently perform the next task.

[0512] Step 7:

[0513] The server automatically generates improvement suggestions and sends them as prompt messages to the AI ​​model, providing feedback to the user. The input is a performance report, and the output is a prompt message summarizing the improvement suggestions.

[0514] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0515] This invention provides a system for efficiently automating business processes and enabling operation that takes user emotions into consideration. The system features business flow observation, repetitive task detection, process automation (RPA), improvement suggestions, and the provision of an interactive agent. Furthermore, by incorporating an emotion engine, it enables interaction based on the user's emotional state.

[0516] This system primarily operates around servers, terminals, and users. First, when users perform their daily tasks, terminals record operation logs and transfer them to the server. The server uses this data to analyze common patterns in the workflow. Through this analysis, repetitive tasks are detected. The server then uses generative models to create detailed blueprints for these repetitive tasks and automates the business processes by directly importing them into an RPA tool.

[0517] After automating a business process, the server monitors the process and analyzes performance data. Based on the detected problems and potential improvements, it generates improvement suggestions and proposes them to the user. The user can review these suggestions on their terminal and provide feedback as needed.

[0518] Another feature of this invention is that an emotion engine is integrated into the server. This emotion engine detects the user's emotional state in real time during communication and adjusts the conversational agent's response based on that emotion. This provides users with a more user-friendly and effective interaction.

[0519] As a concrete example, consider call center operations. When an operator uses a terminal to input customer information and responds to inquiries, the server observes the operator's actions and identifies repetitive tasks. By introducing automated processes, operators are freed from repetitive input tasks and can concentrate on solving more complex problems. Furthermore, if the conversational agent detects the operator's stress through an emotion engine, it modifies its response to support the work. In this way, the system of the present invention provides an effective solution for simultaneously achieving efficiency and user comfort.

[0520] The following describes the processing flow.

[0521] Step 1:

[0522] The terminal monitors and records all operation logs and data inputs as the user performs their work activities. This information is prepared to be sent to the server in real time, as it forms the basis for subsequent analysis.

[0523] Step 2:

[0524] The server receives operation log data sent from terminals and centrally stores it in a database. Machine learning algorithms are used to analyze recurring patterns in the data and identify repetitive tasks.

[0525] Step 3:

[0526] The server generates a detailed blueprint of the business process, including repetitive tasks, based on the analysis results. This blueprint is created through a generative model and converted into a format acceptable to the RPA tool.

[0527] Step 4:

[0528] Users review the design drawings generated from the server via their terminals and evaluate whether their contents meet business requirements. If necessary, they provide feedback on the design drawings to the server, which then makes corrections.

[0529] Step 5:

[0530] The server imports the received blueprint into the RPA tool and starts the automation process. This process automates the identified repetitive tasks and executes them under the server's control.

[0531] Step 6:

[0532] The server monitors the running automation process in real time and analyzes the collected performance data. Based on this analysis, it determines the need for further improvements and generates optimization proposals.

[0533] Step 7:

[0534] When a user communicates with the server through an interactive agent, the emotion engine analyzes the user's emotional state. Based on this analysis, the server selects an appropriate communication style and provides responses that reduce the user's psychological burden.

[0535] (Example 2)

[0536] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0537] There is a need to simultaneously achieve efficiency improvements in business processes and enhance the quality of user interaction. While existing systems are automating business processes, they lack consideration for the emotional state of users, making it difficult to balance business efficiency and user satisfaction.

[0538] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0539] In this invention, the server includes means for monitoring business processes and identifying repetitive tasks, means for designing business processes using a generative artificial intelligence model, and means for recognizing emotional states and adjusting the responses of the dialogue system. This enables improved efficiency in business processes and effective information exchange that takes into account the user's emotions.

[0540] "Business processes" refer to a series of tasks and procedures performed within an organization or business, carried out to achieve a specific goal.

[0541] "Repetitive tasks" are those that involve repeating the same operations or procedures many times, and are usually expected to become more efficient through automation.

[0542] A "generative artificial intelligence model" is an AI technology that has the function of generating information using digital data and algorithms, and is used to optimize business processes.

[0543] A "business process" refers to the steps and flows necessary to carry out a business task, and is designed and adjusted for efficient execution.

[0544] A "task automation system" refers to a process that uses software or machines to streamline or eliminate manual work performed by humans.

[0545] "Information exchange" refers to the activity of transmitting and sharing information between people or systems, and is carried out for the purpose of mutual understanding and decision-making.

[0546] A "dialogue system" is software that enables communication between humans and computers, and generally uses speech recognition or natural language processing.

[0547] "Emotional state" refers to an individual's psychological condition and emotions, and is usually described using indicators that include stress levels and satisfaction levels.

[0548] This invention is a system that streamlines business processes and enables user-centric interactions, primarily involving servers, terminals, and users. Specific embodiments are described below.

[0549] The server collects business process information by monitoring the operations performed by users on their terminals. The terminals record a series of operation logs and transfer this data to the server. The server uses this data to analyze the business flow and identify repetitive tasks using specialized data analysis software, such as Python or R.

[0550] Based on the information from these iterative tasks, the server uses a generative artificial intelligence model to design the business process. This generative AI model may utilize open-source platforms such as TensorFlow or PyTorch. The generated design information is then imported into an automation system, such as UiPath or Automation Anywhere, to automate the business process.

[0551] Furthermore, the server incorporates sentiment analysis capabilities, collecting user interactions and analyzing the user's emotional state in real time. For example, if the server determines that the user is stressed, it adjusts its response through the dialogue system to provide a more user-friendly environment.

[0552] As a concrete example, in call center operations, when an operator inputs customer data on a terminal and responds to an inquiry, the server monitors the operation and automates repetitive tasks. As a result, operators can dedicate more time to handling more complex inquiries. In addition, an emotion engine can detect the operator's stress level and adjust the response accordingly, thereby improving the work environment.

[0553] An example of a prompt for a generated AI model is, "Detect a specific business procedure and create a detailed blueprint for automating it." This allows the server to provide information that helps streamline the given procedure.

[0554] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0555] Step 1:

[0556] When a user performs a task, the terminal records each operation in real time. Input includes user operation data (e.g., button clicks, input field data). Output is an operation log file, which details what and how the user performed the operation. This log file later serves as the basis for data analysis.

[0557] Step 2:

[0558] The terminal transfers operation logs recorded at regular intervals to the server. The input is an operation log file stored on the terminal. The output is business process log data stored in the server's database. This data is used for future data analysis on the server.

[0559] Step 3:

[0560] The server analyzes operation logs in the database to identify business processes. In particular, it utilizes a generative AI model to find patterns in repetitive tasks. The input is all operation logs stored in the database. The output is a list of identified business processes and repetitive tasks, which serves as the foundation for proceeding to the next automation step.

[0561] Step 4:

[0562] The server uses a generative AI model to create a blueprint for repetitive tasks. The input is the business flow data identified in step 3. The output is a detailed blueprint of the automatable business process, which is used to generate a template for input into the task automation system.

[0563] Step 5:

[0564] The server inputs the design blueprint into the automation system. In this case, an automation tool such as UiPath is used. The input is the created design blueprint template. The output is the program that executes the automated business process, which reduces manual work and improves efficiency.

[0565] Step 6:

[0566] The server continuously monitors automated business processes and collects performance data. Its input is real-time data obtained from running business processes. The output is a business performance report, which is used to identify problems and propose improvements.

[0567] Step 7:

[0568] The server uses an emotion engine to evaluate the user's emotional state during interactions and adjust the dialogue system's response accordingly. The input is user interaction data. The output is an adjusted response message, enabling dialogue that adapts to the user's emotions.

[0569] (Application Example 2)

[0570] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0571] In brick-and-mortar retail operations, there is a need to improve the efficiency of repetitive tasks performed by staff while providing flexible, emotion-based support. However, current systems are insufficient in automating processes that take staff emotional states into account and in adjusting support through conversational agents, which can lead to decreased efficiency and a decline in the quality of customer service.

[0572] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0573] In this invention, the server includes means for observing work activities and detecting repetitive tasks, means for creating process blueprints using generative models, and means for detecting the emotional state of workers and adjusting instructions and support content based on that state. This enables store staff to receive emotionally responsive support while improving operational efficiency through process automation.

[0574] "Business activities" refers to the collective work and processes that an organization or individual routinely undertakes to achieve a predetermined objective.

[0575] "Repetitive tasks" are those that involve performing the same procedure or task multiple times, and are usually tasks that can be automated.

[0576] A "generative model" is an algorithm or method for generating new information or structures based on data.

[0577] A "process blueprint" is a visual or descriptive representation of a business process that includes the details necessary for process automation.

[0578] "Process automation tools" are applications and platforms used to automate tasks performed by humans using machines or software.

[0579] "Monitoring" is the process of observing a series of activities or systems, and evaluating and recording their behavior and state.

[0580] An "improvement suggestion" is a proposal or opinion aimed at improving the efficiency and effectiveness of the current system or process.

[0581] A "conversational agent that facilitates communication" is a system or program designed to interact with users and promote smooth communication.

[0582] "Detecting an emotional state" means analyzing an individual's current emotions and identifying that state.

[0583] "Adjusting support content" means changing the type and method of support provided to an individual based on the information detected.

[0584] This application example is a system that enables improved operational efficiency and emotion-based staff support in physical stores. The server receives log data to record work activities and analyzes it to identify repetitive tasks. Based on the collected data, a generative model is used to create process blueprints, which are then imported into UiPath, a process automation tool. This automation process reduces the workload on staff.

[0585] Furthermore, the terminal uses an emotion detection algorithm based on TensorFlow to analyze the staff's emotional state in real time. Based on the analysis results, the server adjusts the instructions and support provided to the staff and notifies the store staff through an interactive agent that facilitates communication. This allows staff to receive appropriate support according to their emotional state.

[0586] For example, when store staff register product information several times a day, the terminal records this activity and adds it to an automated process on the server. Furthermore, if staff fatigue is detected, the terminal is notified with suggestions for breaks or, if possible, instructions to reassign the current task to another staff member.

[0587] An example of a prompt to input into the generating AI model is: "Manage repetitive tasks in an automated manner, taking into account the emotions of the staff. Suggest appropriate responses when staff are experiencing high levels of fatigue." This prompt is expected to enable the system to generate appropriate action plans, balancing operational efficiency with staff well-being.

[0588] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0589] Step 1:

[0590] The terminal records staff operation logs and sends them to the server. The input is data of the daily work operations performed by each staff member. The log data includes frequently performed operations, and this data is analyzed on the server side and used to identify repetitive tasks as common patterns. The output is operation log data expressed in a format suitable for analysis.

[0591] Step 2:

[0592] The server analyzes the received log data and identifies repetitive tasks. The input is log data sent from the terminal, and the output is a list of identified repetitive tasks. In this step, a generative AI model is used to analyze the data and extract tasks suitable for process automation.

[0593] Step 3:

[0594] The server leverages a generative model to create blueprints for repetitive tasks and converts them into a format that can be imported into process automation tools. The input is a list of identified repetitive tasks, and the output is blueprint data in a format suitable for process automation tools. This blueprint is then converted into a process that can be executed by subsequent process automation tools (e.g., UiPath).

[0595] Step 4:

[0596] The server uses a TensorFlow-based algorithm to receive and analyze staff emotional states from terminals in real time. The input is emotional data based on staff biometric information and actions, while the output is analyzed emotional state data. In this step, emotions are estimated using actual biometric data, and the results are used to support staff.

[0597] Step 5:

[0598] The server adjusts instructions and support for staff based on their emotional state and notifies them via their terminals. The input is analyzed emotional state data, and the output is emotionally appropriate instructions and support messages. For example, if fatigue is detected, a break suggestion will be displayed on the terminal.

[0599] Step 6:

[0600] Users receive support instructions and check the progress of tasks via their terminals. Inputs are instructions and support messages notified from the server, and outputs are the user's receipt of instructions and feedback. In this step, bidirectional communication between the server and the terminal ensures that tasks and support proceed smoothly.

[0601] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0602] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0603] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0604] [Fourth Embodiment]

[0605] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0606] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0607] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0608] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0609] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0610] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0611] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0612] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0613] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0614] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0615] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0616] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0617] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0618] This invention provides an advanced system for automating corporate business processes, and specific embodiments thereof are shown below.

[0619] The system based on this invention primarily involves three parties: a server, a terminal, and a user. The user uses software on the terminal to perform their work activities. The terminal records the user's operation logs and data input history and transfers them to the server. The server aggregates the received data and analyzes common patterns in the workflow. This is achieved through algorithms utilizing machine learning.

[0620] The analysis automatically detects repetitive tasks and identifies processes that are candidates for RPA automation. These processes are then translated into natural language blueprints using generative models. These blueprints specify the process steps, required data, and outputs. These blueprints are then converted into a format understandable by the RPA tool and imported from the server to the tool. This automates the selected processes.

[0621] The execution of automated processes is continuously monitored by the server, and process performance data is collected and analyzed. This identifies areas that need improvement, and improvement suggestions are presented to the user. The user can review the provided improvement suggestions and provide feedback to the server as needed. This feedback helps to further improve the entire system.

[0622] Furthermore, the server incorporates an interactive agent, enabling real-time communication with users. This allows users to contact the server via their terminal to resolve questions or inquire about suggestions for process improvements.

[0623] A concrete example is a routine data entry task performed on a terminal at a certain company. The server observes this data entry work and identifies that the task is regular and repetitive. A blueprint is generated and imported into an automation tool, automating subsequent data entry tasks. As a result, users can concentrate their efforts on other important tasks.

[0624] Thus, by implementing the system of the present invention, it is possible to improve operational efficiency and reduce costs.

[0625] The following describes the processing flow.

[0626] Step 1:

[0627] The terminal observes the user's work operations in real time. It meticulously records operation logs and data entry timing, and prepares to send this information to the server in a format that can be used for later analysis.

[0628] Step 2:

[0629] The server collects operation log data sent from terminals and begins centralized management. Using machine learning algorithms, it automatically analyzes frequently occurring work patterns and procedures from the data. This extracts repetitive tasks and identifies processes that are candidates for RPA (Robotic Process Automation) implementation.

[0630] Step 3:

[0631] The server uses a generative model to create a blueprint in natural language, based on the details of the business processes identified through analysis. The blueprint clearly specifies the concrete steps, required datasets, and output specifications. This blueprint is then output in a format that RPA tools can accept.

[0632] Step 4:

[0633] The user uses a terminal to review the blueprints provided by the server and checks whether the process content matches their own workflow. They provide feedback on the blueprints, either approving them or requesting necessary modifications, and requesting corrections from the server as needed.

[0634] Step 5:

[0635] The server directly imports the user-approved blueprints into the RPA tool. This conversion is performed using scripted steps, and the automated process begins based on the imported blueprints.

[0636] Step 6:

[0637] The server continuously monitors automated processes in operation. It collects performance data for each process and monitors its operational status. Based on this, it identifies areas where improvement is possible and creates improvement plans based on the analysis results.

[0638] Step 7:

[0639] Users communicate questions and opinions to the server using the chatbot function on their devices. This interactive agent allows users to receive detailed explanations about improvement suggestions, helping them to deepen their understanding of business process automation.

[0640] (Example 1)

[0641] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0642] Traditional business processes often involve repetitive tasks, making efficient automation difficult. Furthermore, process improvement and optimization were limited due to insufficient communication with users. This invention aims to solve these problems and achieve business efficiency and optimization.

[0643] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0644] In this invention, the server includes means for observing business processes and detecting reproducible operations, means for creating a processing plan using a generation algorithm, and means for converting the plan into a format transferable to a processing automation means. This enables efficient automation of business processes and allows for the discovery and immediate response to areas for improvement through smooth communication with users.

[0645] "Business processes" refer to a series of business activities that companies and organizations perform on a daily basis, and these include repetitive tasks such as data entry and document creation.

[0646] "Reproducible operations" refer to routine user actions that are repeated many times under certain conditions, and which can usually be performed manually to achieve similar results.

[0647] A "generative algorithm" refers to a computational method or procedure that extracts specific patterns or features from data to create new information or blueprints.

[0648] A "plan" is a design document that details the procedures, requirements, and outputs for automating a business process.

[0649] "Process automation means" refers to methods and tools that use machines or software to execute business processes without human intervention.

[0650] "Interactive means" refers to means of mutual communication provided for the exchange of information and the receipt of instructions between a user and a system, and usually includes interfaces that use natural language.

[0651] "Information and communication means" refers to communication methods for transmitting data and information to users, and includes email and notification functions.

[0652] "Means of collecting user feedback and contributing to the overall improvement of the system" refers to methods of improving the system's performance and functionality by collecting and analyzing user feedback information.

[0653] The system of this invention is designed to efficiently automate and optimize a company's business processes. This system is primarily composed of a system in which a server, a terminal, and a user cooperate.

[0654] The server uses a high-performance computing system as the hardware necessary for information processing, and as software, it utilizes a "data processing framework" for data processing and "machine learning algorithms" for machine learning. Specifically, it integrates received business data and analyzes patterns based on the aggregated information.

[0655] A terminal is a device used by users to perform their daily tasks, and this includes typical computers and smart devices. Users input work data through the terminal, and this information is transmitted to the server.

[0656] Automated processes are continuously monitored by a server, and their performance data is collected. This collected data is then communicated to the user through the system's interactive mechanisms, providing suggestions for improvement. Users can then provide feedback based on these suggestions. This feedback contributes to further automation and overall system functionality improvements.

[0657] A concrete example is the data entry task that users perform on a daily basis. The server observes this operation, detects reproducible operations, and automatically creates a process plan using a generation algorithm. This plan is implemented in the system using process automation means, and the user's burden is significantly reduced by automating the work.

[0658] An example of a prompt using a generative AI model is, "Observe your daily data entry tasks and identify patterns that can be automated." This prompt allows the server to begin designing an effective automation process.

[0659] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0660] Step 1:

[0661] The terminal records user operation logs and data entry history and sends them to the server. The input includes all of the user's terminal operations. This data is sent to the server as a formatted log file. The server receives this file and stores it in its database.

[0662] Step 2:

[0663] The server aggregates the received operation logs using a "data processing framework." The input is log data sent from the terminal. The data is categorized, and unnecessary information is filtered out. This results in organized log data that provides a comprehensive overview of the business process.

[0664] Step 3:

[0665] The server uses a "machine learning algorithm" to analyze common patterns in business workflows based on aggregated data. Organized logs are used as input data, and patterns of specific business processes are obtained as output. In this step, different algorithms are tested during the analysis process, and the optimal model is selected.

[0666] Step 4:

[0667] The server detects processes that can be automated based on the analysis results and automatically generates process plans using a generative AI model. The input is the pattern analysis results, and the output is a plan that includes business procedures and necessary data. This specific operation is achieved through text generation utilizing natural language processing technology.

[0668] Step 5:

[0669] The server converts the generated plan into a "process automation tool" and formats it into a format that can be imported into the process automation tool. The input is a plan in natural language, and the output is a script or template that can be directly applied to the system. This step involves data transfer via an API.

[0670] Step 6:

[0671] The server monitors the execution of automated processes and continuously collects performance data. Inputs include log data generated during system execution. Outputs are metrics data indicating the efficiency and performance of the processes. This allows for real-time assessment of process health.

[0672] Step 7:

[0673] The server identifies areas for improvement based on the analyzed performance data and notifies the user. The input is the collected metrics data, and the output is recommended improvements. Specifically, it uses a notification system to send a message to the user's device.

[0674] Step 8:

[0675] The user reviews the improvement suggestions received from the server and provides feedback to the server as needed. The input is the received improvement suggestions, and the output is the user's evaluation and change requests. As a concrete example, opinions and suggestions can be sent using the feedback function on the device.

[0676] (Application Example 1)

[0677] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0678] In manufacturing environments, repetitive tasks performed by workers and their efficiency are crucial, but labor shortages and the complexity of the work can hinder efficient production. Furthermore, a lack of proper instructions on work procedures and suggestions for improvement often leads to decreased productivity. To address this, effective automation to support work and accurate, real-time feedback are necessary.

[0679] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0680] In this invention, the server includes means for observing work processes and identifying repetitive steps, means for creating process blueprints using generation algorithms, and means for converting the blueprints into a format that can be imported into process automation software. This enables efficient automation of repetitive steps and presentation of work procedures using a visual aid device.

[0681] "Observing the work process" means monitoring a series of tasks and procedures performed in manufacturing or service industries and evaluating the situation and progress.

[0682] "Recurring process discovery" is the process of identifying and extracting patterns in which similar tasks are performed regularly.

[0683] A "generative algorithm" is a program that uses machine learning or artificial intelligence technology to automatically derive patterns and rules from data.

[0684] "Creating a blueprint for a process" means visualizing or documenting specific work procedures or processes in a way that other systems and software can understand.

[0685] "Process automation software" is a program that automatically executes specific work procedures, minimizing the need for human intervention.

[0686] A "visual assistance device" is a device that provides visual information to human workers, helping them to perform their tasks efficiently and accurately.

[0687] A "generative AI model" is a framework for artificial intelligence that learns from large amounts of data to generate new information and predictions.

[0688] A "prompt sentence" is an input sentence used to give specific instructions or questions to an artificial intelligence model in order to obtain a result.

[0689] To implement this invention, it is necessary to build a system in which a server, a terminal, and a user work together. The server uses a machine learning-based generation algorithm to monitor the business process and identify repetitive steps. The terminal acts as an intermediate device that transmits user operation logs and data acquired from visual devices to the server.

[0690] The server primarily uses Python and TensorFlow to analyze data and create process blueprints. These blueprints are then converted into a format that can be imported into RPA tools. This process automates identified repetitive steps.

[0691] Users can use smart glasses as a visual aid to visually receive work instructions, enabling them to proceed with the next task efficiently and accurately. This visual aid incorporates image processing libraries such as OpenCV, analyzing the work status in real time and providing necessary instructions to the user.

[0692] As a concrete example, the present invention can be applied to a screw-tightening process performed by workers on a manufacturing production line. In this case, the user receives real-time instructions such as "Tighten the next screw within 3 seconds" via smart glasses worn by the user. As a whole system, this improves work efficiency.

[0693] Possible prompts for the generated AI model include instructions such as, "Please provide information to improve the efficiency of the screw tightening process." Using these prompts, the generated AI model provides additional improvement suggestions and detailed procedural instructions to the user via the server.

[0694] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0695] Step 1:

[0696] The terminal collects user operation logs and data obtained from the visual device and sends them to the server. The input here is the user operation log and data from the visual device, which are integrated and converted into packets. Data packets are generated as output and forwarded to the server.

[0697] Step 2:

[0698] The server observes the business process based on the received data packets and identifies specific repetitive steps. It applies machine learning algorithms to extract patterns from the input data. This process outputs a list of repetitive steps.

[0699] Step 3:

[0700] Using a generative algorithm, the server creates a process blueprint. In this step, a machine learning model receives a list of iterative steps as input and generates a process blueprint based on it. The output is the process blueprint.

[0701] Step 4:

[0702] The server converts the generated blueprints into a format that can be imported into process automation software. The input is the blueprint, and the output is a file in a format that the automation tool can read. This conversion process enables subsequent automation.

[0703] Step 5:

[0704] The server's role is to monitor automated processes and collect performance data. It collects data on running processes as input and generates performance reports as output.

[0705] Step 6:

[0706] The user receives real-time work instructions through an interface provided via smart glasses. Input is instruction data received from a server, and output is visual instructions to the user. These instructions enable the worker to efficiently perform the next task.

[0707] Step 7:

[0708] The server automatically generates improvement suggestions and sends them as prompt messages to the AI ​​model, providing feedback to the user. The input is a performance report, and the output is a prompt message summarizing the improvement suggestions.

[0709] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0710] This invention provides a system for efficiently automating business processes and enabling operation that takes user emotions into consideration. The system features business flow observation, repetitive task detection, process automation (RPA), improvement suggestions, and the provision of an interactive agent. Furthermore, by incorporating an emotion engine, it enables interaction based on the user's emotional state.

[0711] This system primarily operates around servers, terminals, and users. First, when users perform their daily tasks, terminals record operation logs and transfer them to the server. The server uses this data to analyze common patterns in the workflow. Through this analysis, repetitive tasks are detected. The server then uses generative models to create detailed blueprints for these repetitive tasks and automates the business processes by directly importing them into an RPA tool.

[0712] After automating a business process, the server monitors the process and analyzes performance data. Based on the detected problems and potential improvements, it generates improvement suggestions and proposes them to the user. The user can review these suggestions on their terminal and provide feedback as needed.

[0713] Another feature of this invention is that an emotion engine is integrated into the server. This emotion engine detects the user's emotional state in real time during communication and adjusts the conversational agent's response based on that emotion. This provides users with a more user-friendly and effective interaction.

[0714] As a concrete example, consider call center operations. When an operator uses a terminal to input customer information and responds to inquiries, the server observes the operator's actions and identifies repetitive tasks. By introducing automated processes, operators are freed from repetitive input tasks and can concentrate on solving more complex problems. Furthermore, if the conversational agent detects the operator's stress through an emotion engine, it modifies its response to support the work. In this way, the system of the present invention provides an effective solution for simultaneously achieving efficiency and user comfort.

[0715] The following describes the processing flow.

[0716] Step 1:

[0717] The terminal monitors and records all operation logs and data inputs as the user performs their work activities. This information is prepared to be sent to the server in real time, as it forms the basis for subsequent analysis.

[0718] Step 2:

[0719] The server receives operation log data sent from terminals and centrally stores it in a database. Machine learning algorithms are used to analyze recurring patterns in the data and identify repetitive tasks.

[0720] Step 3:

[0721] The server generates a detailed blueprint of the business process, including repetitive tasks, based on the analysis results. This blueprint is created through a generative model and converted into a format acceptable to the RPA tool.

[0722] Step 4:

[0723] Users review the design drawings generated from the server via their terminals and evaluate whether their contents meet business requirements. If necessary, they provide feedback on the design drawings to the server, which then makes corrections.

[0724] Step 5:

[0725] The server imports the received blueprint into the RPA tool and starts the automation process. This process automates the identified repetitive tasks and executes them under the server's control.

[0726] Step 6:

[0727] The server monitors the running automation process in real time and analyzes the collected performance data. Based on this analysis, it determines the need for further improvements and generates optimization proposals.

[0728] Step 7:

[0729] When a user communicates with the server through an interactive agent, the emotion engine analyzes the user's emotional state. Based on this analysis, the server selects an appropriate communication style and provides responses that reduce the user's psychological burden.

[0730] (Example 2)

[0731] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0732] There is a need to simultaneously achieve efficiency improvements in business processes and enhance the quality of user interaction. While existing systems are automating business processes, they lack consideration for the emotional state of users, making it difficult to balance business efficiency and user satisfaction.

[0733] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0734] In this invention, the server includes means for monitoring business processes and identifying repetitive tasks, means for designing business processes using a generative artificial intelligence model, and means for recognizing emotional states and adjusting the responses of the dialogue system. This enables improved efficiency in business processes and effective information exchange that takes into account the user's emotions.

[0735] "Business processes" refer to a series of tasks and procedures performed within an organization or business, carried out to achieve a specific goal.

[0736] "Repetitive tasks" are those that involve repeating the same operations or procedures many times, and are usually expected to become more efficient through automation.

[0737] A "generative artificial intelligence model" is an AI technology that has the function of generating information using digital data and algorithms, and is used to optimize business processes.

[0738] A "business process" refers to the steps and flows necessary to carry out a business task, and is designed and adjusted for efficient execution.

[0739] A "task automation system" refers to a process that uses software or machines to streamline or eliminate manual work performed by humans.

[0740] "Information exchange" refers to the activity of transmitting and sharing information between people or systems, and is carried out for the purpose of mutual understanding and decision-making.

[0741] A "dialogue system" is software that enables communication between humans and computers, and generally uses speech recognition or natural language processing.

[0742] "Emotional state" refers to an individual's psychological condition and emotions, and is usually described using indicators that include stress levels and satisfaction levels.

[0743] This invention is a system that streamlines business processes and enables user-centric interactions, primarily involving servers, terminals, and users. Specific embodiments are described below.

[0744] The server collects business process information by monitoring the operations performed by users on their terminals. The terminals record a series of operation logs and transfer this data to the server. The server uses this data to analyze the business flow and identify repetitive tasks using specialized data analysis software, such as Python or R.

[0745] Based on the information from these iterative tasks, the server uses a generative artificial intelligence model to design the business process. This generative AI model may utilize open-source platforms such as TensorFlow or PyTorch. The generated design information is then imported into an automation system, such as UiPath or Automation Anywhere, to automate the business process.

[0746] Furthermore, the server incorporates sentiment analysis capabilities, collecting user interactions and analyzing the user's emotional state in real time. For example, if the server determines that the user is stressed, it adjusts its response through the dialogue system to provide a more user-friendly environment.

[0747] As a concrete example, in call center operations, when an operator inputs customer data on a terminal and responds to an inquiry, the server monitors the operation and automates repetitive tasks. As a result, operators can dedicate more time to handling more complex inquiries. In addition, an emotion engine can detect the operator's stress level and adjust the response accordingly, thereby improving the work environment.

[0748] An example of a prompt for a generated AI model is, "Detect a specific business procedure and create a detailed blueprint for automating it." This allows the server to provide information that helps streamline the given procedure.

[0749] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0750] Step 1:

[0751] When a user performs a task, the terminal records each operation in real time. Input includes user operation data (e.g., button clicks, input field data). Output is an operation log file, which details what and how the user performed the operation. This log file later serves as the basis for data analysis.

[0752] Step 2:

[0753] The terminal transfers operation logs recorded at regular intervals to the server. The input is an operation log file stored on the terminal. The output is business process log data stored in the server's database. This data is used for future data analysis on the server.

[0754] Step 3:

[0755] The server analyzes operation logs in the database to identify business processes. In particular, it utilizes a generative AI model to find patterns in repetitive tasks. The input is all operation logs stored in the database. The output is a list of identified business processes and repetitive tasks, which serves as the foundation for proceeding to the next automation step.

[0756] Step 4:

[0757] The server uses a generative AI model to create a blueprint for repetitive tasks. The input is the business flow data identified in step 3. The output is a detailed blueprint of the automatable business process, which is used to generate a template for input into the task automation system.

[0758] Step 5:

[0759] The server inputs the design blueprint into the automation system. In this case, an automation tool such as UiPath is used. The input is the created design blueprint template. The output is the program that executes the automated business process, which reduces manual work and improves efficiency.

[0760] Step 6:

[0761] The server continuously monitors automated business processes and collects performance data. Its input is real-time data obtained from running business processes. The output is a business performance report, which is used to identify problems and propose improvements.

[0762] Step 7:

[0763] The server uses an emotion engine to evaluate the user's emotional state during interactions and adjust the dialogue system's response accordingly. The input is user interaction data. The output is an adjusted response message, enabling dialogue that adapts to the user's emotions.

[0764] (Application Example 2)

[0765] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0766] In brick-and-mortar retail operations, there is a need to improve the efficiency of repetitive tasks performed by staff while providing flexible, emotion-based support. However, current systems are insufficient in automating processes that take staff emotional states into account and in adjusting support through conversational agents, which can lead to decreased efficiency and a decline in the quality of customer service.

[0767] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0768] In this invention, the server includes means for observing work activities and detecting repetitive tasks, means for creating process blueprints using generative models, and means for detecting the emotional state of workers and adjusting instructions and support content based on that state. This enables store staff to receive emotionally responsive support while improving operational efficiency through process automation.

[0769] "Business activities" refers to the collective work and processes that an organization or individual routinely undertakes to achieve a predetermined objective.

[0770] "Repetitive tasks" are those that involve performing the same procedure or task multiple times, and are usually tasks that can be automated.

[0771] A "generative model" is an algorithm or method for generating new information or structures based on data.

[0772] A "process blueprint" is a visual or descriptive representation of a business process that includes the details necessary for process automation.

[0773] "Process automation tools" are applications and platforms used to automate tasks performed by humans using machines or software.

[0774] "Monitoring" is the process of observing a series of activities or systems, and evaluating and recording their behavior and state.

[0775] An "improvement suggestion" is a proposal or opinion aimed at improving the efficiency and effectiveness of the current system or process.

[0776] A "conversational agent that facilitates communication" is a system or program designed to interact with users and promote smooth communication.

[0777] "Detecting an emotional state" means analyzing an individual's current emotions and identifying that state.

[0778] "Adjusting support content" means changing the type and method of support provided to an individual based on the information detected.

[0779] This application example is a system that enables improved operational efficiency and emotion-based staff support in physical stores. The server receives log data to record work activities and analyzes it to identify repetitive tasks. Based on the collected data, a generative model is used to create process blueprints, which are then imported into UiPath, a process automation tool. This automation process reduces the workload on staff.

[0780] Furthermore, the terminal uses an emotion detection algorithm based on TensorFlow to analyze the staff's emotional state in real time. Based on the analysis results, the server adjusts the instructions and support provided to the staff and notifies the store staff through an interactive agent that facilitates communication. This allows staff to receive appropriate support according to their emotional state.

[0781] For example, when store staff register product information several times a day, the terminal records this activity and adds it to an automated process on the server. Furthermore, if staff fatigue is detected, the terminal is notified with suggestions for breaks or, if possible, instructions to reassign the current task to another staff member.

[0782] An example of a prompt to input into the generating AI model is: "Manage repetitive tasks in an automated manner, taking into account the emotions of the staff. Suggest appropriate responses when staff are experiencing high levels of fatigue." This prompt is expected to enable the system to generate appropriate action plans, balancing operational efficiency with staff well-being.

[0783] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0784] Step 1:

[0785] The terminal records staff operation logs and sends them to the server. The input is data of the daily work operations performed by each staff member. The log data includes frequently performed operations, and this data is analyzed on the server side and used to identify repetitive tasks as common patterns. The output is operation log data expressed in a format suitable for analysis.

[0786] Step 2:

[0787] The server analyzes the received log data and identifies repetitive tasks. The input is log data sent from the terminal, and the output is a list of identified repetitive tasks. In this step, a generative AI model is used to analyze the data and extract tasks suitable for process automation.

[0788] Step 3:

[0789] The server leverages a generative model to create blueprints for repetitive tasks and converts them into a format that can be imported into process automation tools. The input is a list of identified repetitive tasks, and the output is blueprint data in a format suitable for process automation tools. This blueprint is then converted into a process that can be executed by subsequent process automation tools (e.g., UiPath).

[0790] Step 4:

[0791] The server uses a TensorFlow-based algorithm to receive and analyze staff emotional states from terminals in real time. The input is emotional data based on staff biometric information and actions, while the output is analyzed emotional state data. In this step, emotions are estimated using actual biometric data, and the results are used to support staff.

[0792] Step 5:

[0793] The server adjusts instructions and support for staff based on their emotional state and notifies them via their terminals. The input is analyzed emotional state data, and the output is emotionally appropriate instructions and support messages. For example, if fatigue is detected, a break suggestion will be displayed on the terminal.

[0794] Step 6:

[0795] Users receive support instructions and check the progress of tasks via their terminals. Inputs are instructions and support messages notified from the server, and outputs are the user's receipt of instructions and feedback. In this step, bidirectional communication between the server and the terminal ensures that tasks and support proceed smoothly.

[0796] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0797] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0798] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0799] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0800] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0801] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0802] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0803] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0804] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0805] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0806] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0807] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0808] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0809] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0810] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0811] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0812] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0813] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0814] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0815] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0816] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0817] The following is further disclosed regarding the embodiments described above.

[0818] (Claim 1)

[0819] A means of observing the workflow and detecting repetitive tasks,

[0820] A means of creating a process blueprint using a generative model,

[0821] A means of converting design drawings into a format that can be imported into process automation tools,

[0822] A means of monitoring automated processes and identifying areas for improvement,

[0823] A system that includes means of providing an interactive agent to support communication.

[0824] (Claim 2)

[0825] The system according to claim 1, further comprising means for integrating and analyzing business data.

[0826] (Claim 3)

[0827] The system according to claim 1, further comprising a communication means for notifying the user of improvement suggestions.

[0828] "Example 1"

[0829] (Claim 1)

[0830] A means for observing business processes and detecting reproducible operations,

[0831] A means of creating a processing plan diagram using a generation algorithm,

[0832] A means for converting a plan into a format that can be transferred to an automated processing means,

[0833] Means for monitoring automated processes and identifying areas for improvement,

[0834] A means of providing interactive tools to support information exchange,

[0835] A means for accumulating and analyzing processing information,

[0836] A system that includes this.

[0837] (Claim 2)

[0838] The system according to claim 1, further comprising information and communication means for notifying the user of improvements.

[0839] (Claim 3)

[0840] The system according to claim 1, comprising means for collecting responses from users and contributing to the improvement of the overall system functionality.

[0841] "Application Example 1"

[0842] (Claim 1)

[0843] A means of observing the work process and identifying recurring steps,

[0844] A means of creating a process blueprint using a generation algorithm,

[0845] A means of converting design drawings into a format that can be imported into process automation software,

[0846] A means of monitoring automated processes and identifying areas for improvement,

[0847] A means of providing interactive programs that support communication with humans,

[0848] A means of showing procedures to workers using visual aids,

[0849] A system that includes means for analyzing data acquired from a visual device and presenting the next work procedure.

[0850] (Claim 2)

[0851] The system according to claim 1, further comprising a function for analyzing and integrating business information.

[0852] (Claim 3)

[0853] The system according to claim 1, further comprising means for notifying users of improvement suggestions.

[0854] "Example 2 of combining an emotion engine"

[0855] (Claim 1)

[0856] A means of monitoring business processes and identifying repetitive tasks,

[0857] A means of designing business processes using generative artificial intelligence models,

[0858] A means of converting the design into a format that can be imported into a work automation system,

[0859] A means of observing automated business processes and identifying opportunities for improvement,

[0860] A means of providing a dialogue system to assist in information exchange,

[0861] A system that includes means for recognizing emotional states and adjusting the responses of a dialogue system.

[0862] (Claim 2)

[0863] The system according to claim 1, further comprising means for aggregating and analyzing business information.

[0864] (Claim 3)

[0865] The system according to claim 1, further comprising a communication function for conveying improvement suggestions to users.

[0866] "Application example 2 when combining with an emotional engine"

[0867] (Claim 1)

[0868] A means of observing work activities and detecting repetitive tasks,

[0869] A means of creating a process design using a generative model,

[0870] A means of converting design drawings into a format that can be imported into process automation tools,

[0871] A means of monitoring automated processes and identifying areas for improvement,

[0872] A means of providing an interactive agent to support communication,

[0873] A system that includes means for detecting the emotional state of workers and adjusting instructions and support based on that state.

[0874] (Claim 2)

[0875] The system according to claim 1, further comprising means for integrating and analyzing business information.

[0876] (Claim 3)

[0877] The system according to claim 1, further comprising a communication means for notifying users of improvement suggestions. [Explanation of symbols]

[0878] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of observing the workflow and detecting repetitive tasks, A means of creating a process blueprint using a generative model, A means of converting design drawings into a format that can be imported into process automation tools, A means of monitoring automated processes and identifying areas for improvement, A system that includes means of providing an interactive agent to support communication.

2. The system according to claim 1, further comprising means for integrating and analyzing business data.

3. The system according to claim 1, further comprising a communication means for notifying the user of improvement suggestions.

Images