system

Abstract

Provide a system. 【Solution means】 Means for collecting business data, analyzing the data to visualize the business process, Means for identifying automatable work procedures from the visualized business process, Means for generating an automation plan based on the identified work procedures, Means for importing the generated plan into an automation device and executing the work procedures, Means for monitoring the executed work procedures and generating improvement plans, Means for interacting with users in natural language, Means for improving the management of inbound and outbound goods and delivery arrangements within a logistics facility, A system including the above.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem of the present invention is to reduce the huge costs, time, and psychological burden incurred in the introduction of business automation, and lower the hurdles of RPA introduction faced by corporate management and business improvement personnel. Also, in the midst of the need for continuous improvement and optimization of existing automation processes, the aim is to provide a method for efficiently and flexibly managing and operating business processes.

Means for Solving the Problems

[0005] This invention provides a system that visualizes business flows by collecting and analyzing business data. This system includes a function to identify automatable business processes from the visualized business flows and to generate automation blueprints based on the identified business processes. Furthermore, it imports the generated blueprints into an automation tool to execute the business processes, monitors the executed processes, and generates improvement suggestions. As a result, it achieves business process efficiency and continuous improvement, and enables flexible responses through natural language communication with users.

[0006] "Business data" refers to information and records generated from various business activities within an organization, and is managed in a digital format.

[0007] "Analysis" is the process of analyzing collected data to derive useful patterns, relationships, and insights from it.

[0008] A "business process flow" is a model that shows the series of steps and actions necessary to complete a specific task, and is used for efficient business operations.

[0009] An "automatable business process" refers to a business activity that is currently performed manually but can be processed systematically based on specific rules or conditions.

[0010] An "automation blueprint" is a document or digital file that describes the specific steps and structure for automating an identified business process.

[0011] "Monitoring" is the activity of continuously monitoring the operational status and efficiency of executed business processes and evaluating their performance.

[0012] An "improvement proposal" is a specific suggestion or measure to make business processes more efficient and effective.

[0013] "Natural language communication" is a method of exchanging information using the language that users normally use when interacting with a system. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention provides a system for efficiently automating business processes. This system consists of multiple computer devices, with servers, terminals, and users working together. The following outlines its operation.

[0036] The server first collects business data from various information systems within the organization. During this process, the server continuously imports necessary data from systems such as email systems, CRM, ERP, and sales support systems using APIs and database connections. The collected data is temporarily stored in a database and becomes the subject of analysis.

[0037] Next, the server analyzes the stored business data. Natural language processing (NLP) techniques and machine learning models are used to automatically visualize the business flow from the data. For example, common data manipulation patterns by multiple employees can be extracted and modeled as typical business procedures.

[0038] Subsequently, the server identifies processes from the visualized workflow that are candidates for automation and can be handled using rule-based methods. This process evaluates priorities based on criteria such as the frequency of repetition of tasks, the complexity of manual work, and the error rate, and then determines which processes to automate.

[0039] For processes deemed suitable for automation, the server generates an automation blueprint. This blueprint records specific business procedures, data movement, and error handling methods, and is in a format that can be directly imported into RPA (Robotic Process Automation) tools.

[0040] Subsequently, the terminal imports the generated design into the RPA tool and begins automating the business process in the user's environment. After execution, the server monitors the process and generates improvement suggestions based on the collected performance data. These improvement suggestions aim to optimize and streamline the process and are used for communication with the user.

[0041] Furthermore, the terminal reports the status of business processes to the user via a natural language interface and provides improvement suggestions and answers to questions. In this way, users can interact with the system intuitively and effectively operate automated business processes.

[0042] As a concrete example, when automating the daily data aggregation process in the sales department, the server analyzes the data input patterns of each sales representative and generates an RPA blueprint to standardize the processing of unique data fields. The terminal then uses this blueprint to automatically generate aggregation reports, achieving efficient operations. This system dynamically optimizes the automation process, significantly reducing the burden on users.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The server uses APIs and database connections to collect business data from various information systems within the organization, and automatically performs tasks such as periodically importing data from email systems, CRM, and ERP.

[0046] Step 2:

[0047] The server stores the collected business data in a database and analyzes the data using natural language processing and machine learning models. In this process, the business flow is automatically visualized from the data, and patterned procedures are modeled.

[0048] Step 3:

[0049] The server evaluates the business workflows identified from the analysis results and identifies processes that can be automated. Criteria for selecting processes include the frequency of repetition, the complexity of manual work, and the error rate.

[0050] Step 4:

[0051] The server generates an automation blueprint for the identified process. This blueprint includes specific business procedures, data movement methods, and error handling, and is in a format that can be imported into an RPA tool.

[0052] Step 5:

[0053] The terminal imports the blueprint generated by the server into the RPA tool and completes the necessary settings to start the automated business process. It then launches the bot in the execution environment.

[0054] Step 6:

[0055] Users can check the progress and results of ongoing business processes through their terminals and take timely action if problems arise. This ensures that the effectiveness of the processes in the real world is maintained.

[0056] Step 7:

[0057] The server continuously monitors performance data of executed business processes and records information on efficiency and failures. Based on this, it generates improvement suggestions for further optimization.

[0058] Step 8:

[0059] The terminal provides process status reports and answers user questions via a natural language interface, offering improvement suggestions. Through this interaction, users can gain concrete insights into improving process operations.

[0060] (Example 1)

[0061] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0062] In today's business environment, there is a demand for collecting necessary business information from vast amounts of information resources and analyzing it effectively. Furthermore, as business procedures are increasingly automated based on the analysis results, existing methods may not be sufficient. To address this challenge, it is necessary to develop a system that efficiently automates business procedures and effectively monitors and improves the execution process.

[0063] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0064] In this invention, the server includes means for collecting business information from information resources and analyzing that information to visualize business procedures; means for identifying business procedures that can be automated from the visualized business procedures; and means for incorporating the generated plan into automation technology and executing the business procedures. This makes it possible to efficiently advance the automation of business procedures and to easily monitor and improve them.

[0065] "Information resources" is a general term for multiple databases, applications, and systems that provide business information.

[0066] "Business information" refers to the data and knowledge necessary to perform or manage business operations within an organization.

[0067] "Analysis" is the process of processing collected business information and extracting meaningful patterns and trends.

[0068] "Business procedures" refer to a series of operations and steps necessary to efficiently carry out a business task.

[0069] "Visualization" refers to displaying extracted business procedures and data in a format that is easy for users to understand.

[0070] An "automated procedure" refers to a set of processes or operations designed to automate manual human work.

[0071] "Automation technology" refers to all technologies used to automate business procedures.

[0072] "Monitoring" refers to activities aimed at evaluating the performance of ongoing business procedures and processes and making improvements as needed.

[0073] A "proposal for improvement" refers to specific changes proposed to enhance the efficiency and effectiveness of business procedures and processes.

[0074] "Users" refer to those who operate the system or use the information obtained to perform their duties.

[0075] "Natural language dialogue" refers to providing an interface that uses everyday language so that users can intuitively communicate with the system.

[0076] The invention will now be described in terms of its implementation. This system works in which a server, terminals, and users collaborate to efficiently collect and analyze business information and automate business procedures.

[0077] The server first collects business information from information resources. To do this, the server uses APIs and database connections to retrieve business-related information from various management systems. Standard database management systems and interfaces such as REST APIs are used for information retrieval. The collected information is centrally stored in a database, preparing it for the next analysis step.

[0078] Next, the server analyzes the stored information using natural language processing techniques and machine learning models. Programming languages ​​such as Python and R, along with related libraries like TENSORFLOW® and NLTK, are used. The analysis results are visually represented, providing a flowchart that is easily understandable to the user.

[0079] After completing the analysis and visualization, the server identifies business procedures that can be automated and generates an automation plan based on them. This plan describes the detailed steps of the business procedures and is in a format that can be imported into an RPA tool. The server creates the plan using specific RPA software (e.g., UiPath, Blue Prism).

[0080] The terminal automates business procedures by importing the generated plan into the RPA tool. The terminal reports the progress of the business procedures to the server, which monitors its performance and provides suggestions for improvement.

[0081] Users can interact with the system via a natural language interface and receive notifications about the status of automated tasks. They can also receive immediate answers to questions and requests. Generative AI models are used for system interaction, enabling natural and effective communication.

[0082] As a concrete example, consider data aggregation work in the sales department. This system analyzes the data entry patterns of each employee and generates an RPA plan for standardizing unique data fields. The terminal then uses this plan to automatically create aggregation reports, thereby improving operational efficiency.

[0083] An example of a prompt message would be, "Analyze the update patterns of the sales data and generate an RPA blueprint." By entering this prompt into the system, the server can begin the necessary analysis and blueprint generation.

[0084] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0085] Step 1:

[0086] The server collects business information from information resources. Using API keys and database connection information as input, it retrieves business-related data from email, customer relationship management systems, and corporate resource planning systems. The output is a database containing the raw data. This step converts the data into a consistent format according to a protocol, enabling efficient processing in subsequent analysis steps.

[0087] Step 2:

[0088] The server organizes the collected data and analyzes it using natural language processing (NLP) techniques. It uses the raw data obtained in the previous step as input, performing data processing such as noise reduction, tokenization, and summarization. The output is an analysis showing patterns and trends in business procedures. This analysis extracts significant information from the data using NLP libraries in Python or R.

[0089] Step 3:

[0090] The server visualizes business procedures based on the analysis results. The input is the analysis results from step 2. As output, flowcharts and dashboards are generated, presenting business procedures in a format that is easy for users to understand. As a tool, data visualization software is used to generate the visual information.

[0091] Step 4:

[0092] The server identifies parts of the business process that can be automated and creates an automation plan. The input is a visualized business process. The output is a plan containing detailed steps required for automation, in a format that can be imported into an RPA tool. This step uses algorithms to detect repetitive tasks and manual errors in the business flow and evaluate their priority.

[0093] Step 5:

[0094] The terminal imports the automation plan received from the server into the RPA tool and automates business procedures. The input is the automation plan. The output is the result of the executed automated tasks. Here, automated work is performed according to the plan through the RPA software.

[0095] Step 6:

[0096] The server monitors executed business procedures and collects performance data. The input is execution data obtained from the RPA tool. The output is a monitoring report containing metrics for business improvement. This provides a foundation for evaluating process efficiency and effectiveness and generating improvement proposals.

[0097] Step 7:

[0098] The user receives status reports and improvement suggestions for business processes from the server via a natural language interface. Input includes questions and confirmation requests from the user. Output provides the user with process status information and specific improvement suggestions. The use of a generative AI model enables natural and intuitive dialogue.

[0099] (Application Example 1)

[0100] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0101] Modern logistics facilities suffer from inefficiency and a high risk of human error due to the significant amount of manual work involved in managing the inbound and outbound movement of goods and arranging deliveries. Repetitive tasks, in particular, place a heavy burden on workers and contribute to overall operational efficiency. Solving this problem and improving efficiency within logistics facilities is therefore essential.

[0102] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0103] In this invention, the server includes means for collecting business data, analyzing that data to visualize business processes, identifying work procedures that can be automated from the visualized business processes, and generating an automation plan based on the identified work procedures. This enables efficient management of goods receiving and shipping and delivery arrangements within logistics facilities, reduces human labor, and allows for highly operationally efficient business processes.

[0104] "Business data" refers to information such as records of goods being received and shipped from logistics facilities, inventory information, and shipping instructions.

[0105] "Analysis" is the process of analyzing work patterns based on collected business data to highlight efficient workflows.

[0106] "Visualization" means presenting analysis results visually and providing them in a way that is easy for stakeholders to understand.

[0107] "Work procedure" refers to a series of specific operations or steps involved in performing a task.

[0108] An "automation plan diagram" is a drawing that describes the design for automating specific work procedures in an efficient and error-free manner.

[0109] "Automated equipment" is a general term for hardware and software used to automatically execute specific business procedures.

[0110] "Monitoring" is a process that tracks the status of ongoing work procedures and business processes in real time, enabling a quick response if problems occur.

[0111] An "improvement proposal" is a suggestion for changes or improvements to business processes based on collected monitoring data, aimed at improving their efficiency and accuracy.

[0112] "User" refers to the staff or managers of a logistics facility who use this system for their operations.

[0113] "Dialogue" is the process by which a user and a system interact and exchange information, either verbally or through text.

[0114] "Efficiency improvement" refers to making tasks possible with less time and resources.

[0115] The present invention provides an automation program aimed at improving the efficiency of operations in logistics facilities. The server uses APIs and database connections to collect operational data from various information sources within the logistics facility. This information includes inbound and outbound records, inventory information, and shipping instructions. This data is temporarily stored in a database and undergoes a pre-processing stage for analysis.

[0116] The server analyzes stored data using natural language processing (NLP) and machine learning models to visualize efficient business processes. For example, it automatically extracts the optimal work procedures for smooth receiving and shipping operations. Based on the work procedures identified through the analysis, an automation plan is generated, and the specific work flow is executed by an RPA tool. In this case, UiPath is used as the specific RPA tool.

[0117] The terminal monitors ongoing work procedures in real time and receives improvement suggestions from the server to enhance efficiency and accuracy. Users then adjust their work based on these suggestions. The terminal may utilize Oracle or MySQL® as its monitoring system. Users can also check the status of business processes and inquire about improvements through a natural language interaction interface. This entire process is supported by cloud-based support on the AWS® platform.

[0118] As a concrete example, when new home appliances arrive at a logistics facility, their barcodes are automatically scanned, and a database in the cloud is updated. Next, shipping instructions are automatically generated and sent to the delivery company. Workers can monitor this entire process via their smartphones and intervene with human judgment as needed.

[0119] An example of a prompt for a generating AI model is: "Please explain how to improve process efficiency using natural language processing and RPA in an automated shipping management system for a logistics center. Please include specific operating procedures and areas for improvement." This prompt allows the system to analyze the information and provide it to the user.

[0120] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0121] Step 1:

[0122] The server collects operational data from information systems within the logistics facility. As input, it retrieves data such as inbound / outbound records, inventory information, and shipping instructions using APIs and database connections. This data is temporarily stored in an SQL database to prepare it for subsequent processing.

[0123] Step 2:

[0124] The server analyzes stored data using natural language processing (NLP) techniques and machine learning models. The input is stored business data, and the output is visualization data representing efficient business processes. Specifically, pattern recognition is performed on each dataset to extract necessary features.

[0125] Step 3:

[0126] The server identifies automatable work procedures from the analyzed data and generates an automation plan. The input is visualized data, and the output is an automation plan that can be executed with an RPA tool. At this stage, it determines which procedures are best suited for automation based on the number of repetitions and frequency of each procedure.

[0127] Step 4:

[0128] The terminal imports the generated automation plan into the RPA tool (UiPath) and executes the automated work procedure. The input is the automation plan, and the output is the result of the performed work. Specifically, the automated process is initiated based on the plan, and the work procedure is carried out through a robotic process.

[0129] Step 5:

[0130] The system monitors the work procedures performed by the terminal in real time, and the server analyzes the collected monitoring data to generate improvement suggestions. The input is data recording the progress of the work, and the output is improvement suggestions for efficiency. In actual operation, system logs and execution performance are evaluated, and continuous optimization is performed.

[0131] Step 6:

[0132] Users utilize a natural language dialogue interface via their devices to understand the status of business processes and consider improvement proposals as needed. Input consists of user inquiries made through the dialogue interface, while output is information and suggestions related to the business. Specifically, users use smartphones or PCs to check the progress of their work and apply suggestions from the system.

[0133] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0134] This invention improves the quality of the user interface and supports more effective business efficiency by incorporating an emotion engine into business automation systems. This system optimizes business processes through interaction between servers, terminals, and users.

[0135] The server first collects business data from the organization's information systems and visualizes the business flow by analyzing that data. The business data analyzed by the server is processed to ensure that it is reflected in the business flow without any omissions. Then, it identifies processes that are suitable for automation from the visualized business flow. For the identified processes, the server generates an automation blueprint. This blueprint describes the business processing procedures, data flow, and error handling in detail, and can be imported into RPA tools.

[0136] The terminal uses this automation blueprint to execute business processes. The executed processes are monitored by the server, and performance data is used to generate improvement suggestions. This enables continuous improvement of business processes.

[0137] The emotion engine analyzes the user's emotional state in real time. The device uses the emotion engine to proactively recognize the user's emotions during communication and reflects the results in its service response. For example, if the device detects that the user is stressed, it will be configured to provide a more considerate response. Emotional data is also used to optimize the overall system's responsiveness and efficiency.

[0138] When users use the system, they can interact with servers and terminals through a natural language interface. At this time, insights based on the analysis results of the sentiment engine are provided, including improvement suggestions for user inquiries. This feature allows users to receive more emotionally resonant feedback on their work processes.

[0139] As a concrete example, in a customer support department that handles customer interactions, a system that detects user emotions can help operators determine the priority of their requests. Furthermore, for operators experiencing stress, the system can suggest automating processes to reduce their workload, thereby improving operational efficiency. This system enables increased productivity across the entire organization.

[0140] The following describes the processing flow.

[0141] Step 1:

[0142] The server collects business data from various information systems within the company. Using APIs and database connections, it efficiently imports data from systems such as email, CRM, and ERP, and stores it in an organized database.

[0143] Step 2:

[0144] The server analyzes collected business data and visualizes the business flow. It utilizes natural language processing technology and machine learning models to clearly define business processes and operation patterns, generating a visually represented model.

[0145] Step 3:

[0146] The server identifies processes suitable for automation from the visualized workflow. It selects processes with a high potential for efficiency improvement by considering specific criteria such as the frequency of repetitive tasks, the amount of manual operation, and the error rate.

[0147] Step 4:

[0148] The server generates an automation blueprint based on the identified business process. The blueprint includes procedures, data flow, and error handling, and is output in a format that can be imported into an RPA tool.

[0149] Step 5:

[0150] The terminal imports the blueprint generated by the server into the RPA tool and starts the automated execution of business processes. The configured bot performs a specific task and records the results in the database.

[0151] Step 6:

[0152] The server monitors the executed business processes and collects performance data. It monitors metrics such as execution speed, error rate, and success rate, and generates improvement suggestions based on this data.

[0153] Step 7:

[0154] The terminal communicates with the user through a natural language interface, providing process status reports and suggesting improvements. It also provides real-time answers to user questions and inquiries.

[0155] Step 8:

[0156] A device equipped with an emotion engine analyzes the user's emotional state. It extracts emotional data from communication with the user and uses it to improve service responses and suggest improvements, thereby enhancing the quality of the interaction.

[0157] Step 9:

[0158] Users can monitor the progress and results of business processes and discover opportunities for further business improvement through insights gained from the emotion engine. They can also propose new automated processes as needed.

[0159] (Example 2)

[0160] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0161] In today's business environment, there is a demand for both efficient business processes and user-friendliness. In particular, improving the quality of emotion-based interfaces during the process of automating business processes is a challenge. Furthermore, there is the difficulty of providing individually optimized services that meet the diverse needs of users.

[0162] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0163] In this invention, the server includes means for collecting business data, analyzing that data to visualize business processes, identifying tasks that can be automated from the visualized business processes, and generating automation designs based on the identified tasks. This improves the efficiency of operations and enables the provision of a flexible interface based on the user's emotional state.

[0164] "Business data" refers to information related to business processes obtained from information systems within an organization.

[0165] "Analysis" is the act of processing and analyzing collected data to extract useful information and patterns.

[0166] A "business process" refers to a series of procedures and activities necessary to perform a specific task.

[0167] "Visualization" is the act of representing data and information visually to make them easier to understand.

[0168] "Automated tasks" are those tasks performed by humans that can be made more efficient by applying automation technology.

[0169] "Automation design" is a detailed description of the procedures and methods for executing a process that can be automated.

[0170] A "process automation tool" is software used to automatically execute and manage business tasks.

[0171] "Monitoring" is the activity of continuously monitoring the operating status of a system or process and collecting data.

[0172] "Emotional analysis" is a technology that objectively analyzes a user's emotional state and represents that state using numerical values ​​or categories.

[0173] "Natural language dialogue" refers to communication between a system and a user using the language that humans use on a daily basis.

[0174] This invention is a system for streamlining business processes and improving user interaction. The main components of this system include servers, terminals, and users.

[0175] The server first collects business data from the organization's information systems. It uses database software to issue SQL queries and retrieve the necessary data. Next, it analyzes the data using Python libraries such as Pandas and NumPy to extract information for visualizing business processes. Based on the information obtained from this analysis, it creates flowcharts using visualization tools such as D3.js and Tableau.

[0176] The terminal performs automation design based on visualization data sent from the server. Specifically, it automates business tasks using process automation tools such as UiPath and Automation Anywhere. During the execution of automated tasks, it monitors and collects performance data using Prometheus. This data is visualized in a dashboard format using tools such as Grafana and used to improve business processes.

[0177] Furthermore, the device is equipped with emotion analysis capabilities, using image analysis and natural language processing technologies to analyze the user's emotions in real time. This process utilizes emotion analysis tools from Affectiva and IBM. By recognizing the user's emotional state, the device can provide more human-like and considerate responses.

[0178] Users interact with the system through a natural language interface. In response to user inquiries, the system provides appropriate feedback using prompts generated by a generative AI model. For example, if a user enters the request, "Please check the progress of the current task," the system analyzes real-time business data and reports its progress.

[0179] This system will improve the overall operational efficiency of the organization and enable it to provide users with services tailored to their individual needs.

[0180] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0181] Step 1:

[0182] The server collects business data from the organization's information systems. It uses raw business data obtained from databases as input. This data includes customer information, transaction history, inventory status, etc. SQL queries are executed to extract the data, which then becomes available as output within the server.

[0183] Step 2:

[0184] The server analyzes the collected business data to visualize business processes. The input is the business data collected in Step 1. This data is then statistically analyzed and cleansed using Python's Pandas and NumPy to gain new insights. Based on these insights, a business flowchart is created using visualization tools such as D3.js, and the visualized data is output.

[0185] Step 3:

[0186] The server identifies tasks that can be automated using visualization data. The input is the business flowchart generated in step 2. Through machine learning algorithms, it identifies repetitive tasks that are particularly likely to benefit from automation and outputs the results as an automation design.

[0187] Step 4:

[0188] The terminal automates business tasks based on automation designs provided by the server. It receives automation designs as input and uses UiPath to create automation scripts. For example, it might perform a task to automatically generate a standardized report at a fixed time each day, and output whether the task was completed on schedule.

[0189] Step 5:

[0190] The server monitors tasks performed on terminals and collects performance data. It uses logs and execution result data generated by UiPath as input. Prometheus is used to collect this data in real time, and Grafana is used to visualize it, yielding output that shows efficiency and changes in execution status.

[0191] Step 6:

[0192] The terminal performs emotion analysis during user interaction. Inputs include audio data and webcam footage from user conversations. Using technologies from Affectiva and IBM, it analyzes emotions, generates responses based on that data, and provides user-adapted services as output.

[0193] Step 7:

[0194] Users interact with the system using natural language and receive feedback. They input questions and commands in natural language using prompts generated by a generative AI model. The system analyzes these requests, generates responses based on monitoring and sentiment analysis data, and provides them to the user as output. For example, inputting a request such as "Please check the progress of the current task" will provide progress information.

[0195] (Application Example 2)

[0196] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0197] In modern information processing systems, improving operational efficiency and enhancing user experience are crucial challenges. In particular, there is a demand for services that respond to user emotions and individual needs. However, conventional systems lacked sufficient emotion analysis capabilities, making effective communication with users difficult. This resulted in increased workload and inconsistent service quality.

[0198] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0199] In this invention, the server includes means for collecting business data and analyzing that data to visualize the business flow; means for identifying business processes that can be automated from the visualized business flow; means for generating automation blueprints based on the identified business processes; means for importing the generated blueprints into an automation tool and executing the business processes; means for monitoring the executed business processes and generating improvement suggestions; and means for analyzing the user's emotional state and adjusting the response, and for providing individualized responses based on the user's emotional analysis. This enables efficient execution of business processes that take into account the user's emotional state and personalized user responses.

[0200] "Business data" refers to a collection of information and figures related to business processes generated within an organization.

[0201] A "business process flow" is a visual representation of how business processes progress and how data flows.

[0202] An "automation blueprint" is a design document that describes the specific steps and data flow required to automate an identified business process.

[0203] "Automation tools" is a general term for software or systems used to execute automation blueprints.

[0204] "Monitoring" is the act of observing and recording the operation and performance of a business process in progress.

[0205] An "improvement suggestion" is a specific proposal to improve the efficiency and quality of business processes based on monitoring data.

[0206] "Natural language communication" refers to dialogue and information exchange that takes place through the language that humans normally use.

[0207] "Emotional state" refers to the psychological state a user experiences emotionally, and includes changes in emotions that occur under specific circumstances.

[0208] "Individualized support" refers to providing specific support or services tailored to the individual needs and circumstances of each user.

[0209] This invention aims to improve the user experience by incorporating sentiment analysis functionality into a business automation system. The system operates between a server, a terminal, and a user, with each party's role clearly defined.

[0210] First, the server collects business data from the organization's information systems and stores it in a database. The stored business data is analyzed and visualized as a business flow. Data analysis tools (e.g., Tableau or Power BI) can be used as the software for this analysis. Visualization helps identify which business processes are suitable for automation and generates automation blueprints suitable for import into RPA (Robotic Process Automation) tools. These blueprints include process steps, data flow, and error handling.

[0211] The terminal executes business processes based on the generated blueprints. This process utilizes RPA tools installed on the terminal (e.g., UiPath or Automation Anywhere). During process execution, the server monitors its operation and collects performance data. Based on this data, the system generates process improvement suggestions.

[0212] Communication with users is conducted using natural language processing technologies (e.g., NLTK and spaCy). The terminal also uses an emotion analysis engine (e.g., Emotion API) to analyze the user's emotions in real time. If the user is experiencing stress, the system provides considerate responses from the terminal and offers suggestions to reduce their workload.

[0213] As a concrete example, in a system utilizing a home robot, the robot analyzes the user's emotions from their facial expressions and voice, and if it determines that the user is tired, it recommends relaxation music. An example of a prompt message to the generative AI model used in this case would be: "Program music suggestions for when the user is feeling tired on a Friday night. These suggestions should be based on emotion analysis data."

[0214] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0215] Step 1:

[0216] The server collects business data from the organization's information systems. The information obtained as input includes business process log data and error history. The server stores this data in a database, preparing it for subsequent analysis. The output is a well-organized business dataset.

[0217] Step 2:

[0218] The server visualizes the business flow based on the acquired data. It generates a flowchart from the input dataset and visualizes it using data analysis tools. Specifically, it analyzes the start time, end time, and duration of each business step. The output is a visual map showing the flow of the business process.

[0219] Step 3:

[0220] The server identifies business processes that can be automated from the visualized workflow. Machine learning algorithms are used for analysis, picking out repetitive sections within the process. A workflow map is used as input, and the identified automation candidate processes are output.

[0221] Step 4:

[0222] The server generates automation blueprints based on identified business processes. A template-based approach is used for blueprint generation. Detailed information about the identified processes is used as input data, and the output is a blueprint file for use in automation tools.

[0223] Step 5:

[0224] The terminal imports the automation blueprint into the RPA tool and executes the business process. The business is processed automatically according to the specified procedures. The input is the blueprint file, and the output is the automated business result.

[0225] Step 6:

[0226] The server monitors executed business processes and collects performance data. Monitoring tools are used to obtain data such as execution time and error rates. The input is the execution status of the business processes, and the output is a performance report.

[0227] Step 7:

[0228] The device uses an emotion analysis engine to analyze the user's emotional state in real time. It acquires the user's voice and video as input data and performs calculations to identify their emotional state. The output is the analysis result regarding the user's emotions.

[0229] Step 8:

[0230] The device provides personalized responses based on the user's sentiment analysis results. Based on the analysis, it generates appropriate responses and suggestions and presents them to the user. In this process, the sentiment analysis results are used as input, and the output is specific actions or suggestions.

[0231] Step 9:

[0232] Users interact with the system using natural language communication methods provided by their devices. Appropriate responses and improvement suggestions are provided based on user input. Input is the user's natural language, and output is relevant information and suggestions.

[0233] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0234] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0235] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0236] [Second Embodiment]

[0237] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0238] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0239] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0240] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0241] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0242] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0243] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0244] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0245] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0246] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0247] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0248] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0249] This invention provides a system for efficiently automating business processes. This system consists of multiple computer devices, with servers, terminals, and users working together. The following outlines its operation.

[0250] The server first collects business data from various information systems within the organization. During this process, the server continuously imports necessary data from systems such as email systems, CRM, ERP, and sales support systems using APIs and database connections. The collected data is temporarily stored in a database and becomes the subject of analysis.

[0251] Next, the server analyzes the stored business data. Natural language processing (NLP) techniques and machine learning models are used to automatically visualize the business flow from the data. For example, common data manipulation patterns by multiple employees can be extracted and modeled as typical business procedures.

[0252] Subsequently, the server identifies processes from the visualized workflow that are candidates for automation and can be handled using rule-based methods. This process evaluates priorities based on criteria such as the frequency of repetition of tasks, the complexity of manual work, and the error rate, and then determines which processes to automate.

[0253] For processes deemed suitable for automation, the server generates an automation blueprint. This blueprint records specific business procedures, data movement, and error handling methods, and is in a format that can be directly imported into RPA (Robotic Process Automation) tools.

[0254] Subsequently, the terminal imports the generated design into the RPA tool and begins automating the business process in the user's environment. After execution, the server monitors the process and generates improvement suggestions based on the collected performance data. These improvement suggestions aim to optimize and streamline the process and are used for communication with the user.

[0255] Furthermore, the terminal reports the status of business processes to the user via a natural language interface and provides improvement suggestions and answers to questions. In this way, users can interact with the system intuitively and effectively operate automated business processes.

[0256] As a concrete example, when automating the daily data aggregation process in the sales department, the server analyzes the data input patterns of each sales representative and generates an RPA blueprint to standardize the processing of unique data fields. The terminal then uses this blueprint to automatically generate aggregation reports, achieving efficient operations. This system dynamically optimizes the automation process, significantly reducing the burden on users.

[0257] The following describes the processing flow.

[0258] Step 1:

[0259] The server uses APIs and database connections to collect business data from various information systems within the organization, and automatically performs tasks such as periodically importing data from email systems, CRM, and ERP.

[0260] Step 2:

[0261] The server stores the collected business data in a database and analyzes the data using natural language processing and machine learning models. In this process, the business flow is automatically visualized from the data, and patterned procedures are modeled.

[0262] Step 3:

[0263] The server evaluates the business workflows identified from the analysis results and identifies processes that can be automated. Criteria for selecting processes include the frequency of repetition, the complexity of manual work, and the error rate.

[0264] Step 4:

[0265] The server generates an automation blueprint for the identified process. This blueprint includes specific business procedures, data movement methods, and error handling, and is in a format that can be imported into an RPA tool.

[0266] Step 5:

[0267] The terminal imports the blueprint generated by the server into the RPA tool and completes the necessary settings to start the automated business process. It then launches the bot in the execution environment.

[0268] Step 6:

[0269] Users can check the progress and results of ongoing business processes through their terminals and take timely action if problems arise. This ensures that the effectiveness of the processes in the real world is maintained.

[0270] Step 7:

[0271] The server continuously monitors performance data of executed business processes and records information on efficiency and failures. Based on this, it generates improvement suggestions for further optimization.

[0272] Step 8:

[0273] The terminal provides process status reports and answers user questions via a natural language interface, offering improvement suggestions. Through this interaction, users can gain concrete insights into improving process operations.

[0274] (Example 1)

[0275] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0276] In a modern business environment, it is required to collect necessary business information from a large amount of information resources and effectively analyze it. Furthermore, in the process of automating business procedures based on the analysis results, there may be cases where existing methods cannot adequately handle them. To solve this problem, it is necessary to develop a system that efficiently automates business procedures and effectively monitors and improves the execution process.

[0277] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following respective means.

[0278] In this invention, the server includes means for collecting business information from information resources, analyzing the information to visualize business procedures, means for identifying business procedures that can be automated from the visualized business procedures, and means for incorporating the generated plan into automation technology and executing the business procedures. As a result, it becomes possible to efficiently advance the automation of business procedures and facilitate monitoring and improvement.

[0279] "Information resources" is a general term for a plurality of databases, applications, systems, etc. that provide business information.

[0280] "Business information" is data and knowledge necessary for executing or managing business within an organization.

[0281] "Analysis" is a process of processing the collected business information and extracting meaningful patterns and trends.

[0282] "Business procedures" are a series of operations and procedures necessary for efficiently performing business.

[0283] "Visualization" refers to presenting the extracted business procedures and data in a form that is easy for users to understand.

[0284] "Automation procedures" refer to a series of processes and operations designed to automate human manual work.

[0285] "Automation technology" refers to all technologies used to achieve the automation of business procedures.

[0286] "Monitoring" is an activity for evaluating the performance of ongoing business procedures or processes and making improvements as necessary.

[0287] "Improvement proposal" refers to specific changes proposed to improve the efficiency and effectiveness of business procedures or processes.

[0288] "User" refers to a person who operates the system or utilizes the obtained information to conduct business.

[0289] "Dialogue in natural language" refers to providing an interface using everyday language so that users can intuitively communicate with the system.

[0290] The form for implementing the invention will be described. This system functions through the collaboration of a server, a terminal, and a user to efficiently collect and analyze business information and automate business procedures.

[0291] The server first collects business information from information resources. For this purpose, the server uses API and database connections to obtain information related to business from various management systems. Standard database management systems and interfaces such as REST API are used for information acquisition. The collected information is centrally stored in a database to prepare for the next analysis step.

[0292] Next, the server analyzes the stored information using natural language processing technology and machine learning models. Here, programming languages such as Python and R and related libraries, such as TensorFlow and NLTK, are used. As a result of the analysis, the business procedures are visually represented and provided as a flow in a form that is easily understandable by the user.

[0293] After completing the analysis and visualization, the server identifies business procedures that can be automated and generates an automation plan based on them. This plan describes the detailed steps of the business procedures and is in a format that can be imported into an RPA tool. The server creates the plan using specific RPA software (e.g., UiPath, Blue Prism).

[0294] The terminal automates business procedures by importing the generated plan into the RPA tool. The terminal reports the progress of the business procedures to the server, which monitors its performance and provides suggestions for improvement.

[0295] Users can interact with the system via a natural language interface and receive notifications about the status of automated tasks. They can also receive immediate answers to questions and requests. Generative AI models are used for system interaction, enabling natural and effective communication.

[0296] As a concrete example, consider data aggregation work in the sales department. This system analyzes the data entry patterns of each employee and generates an RPA plan for standardizing unique data fields. The terminal then uses this plan to automatically create aggregation reports, thereby improving operational efficiency.

[0297] An example of a prompt message would be, "Analyze the update patterns of the sales data and generate an RPA blueprint." By entering this prompt into the system, the server can begin the necessary analysis and blueprint generation.

[0298] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0299] Step 1:

[0300] The server collects business information from information resources. As input, it uses an API key and database connection information to extract business-related data from emails, customer relationship management systems, and enterprise resource planning systems. As output, a database storing raw data is obtained. In this step, the data is converted into a consistent format according to the protocol so that it can be efficiently processed in subsequent analysis steps.

[0301] Step 2:

[0302] The server sorts out the collected data and analyzes it using natural language processing technology. As input, it uses the raw data obtained in the previous step to perform data processing such as noise removal, tokenization, and summarization. The output is an analysis result showing the patterns and trends of business procedures. In this analysis, significant information is extracted from the data using NLP libraries in Python and R.

[0303] Step 3:

[0304] The server visualizes business procedures based on the analysis results. The input is the analysis result of Step 2. As output, flowcharts and dashboards are generated, and the business procedures are presented in a form that can be easily understood by users. As a tool, data visualization software is used to generate visual information.

[0305] Step 4:

[0306] The server identifies the parts of business procedures that can be automated and creates an automation plan. The input is the visualized business procedures. The output is a plan describing the detailed procedures required for automation, in a format that can be imported into RPA tools. In this step, an algorithm that detects repetitive tasks and manual errors in the business flow and evaluates priorities is used.

[0307] Step 5:

[0308] The terminal imports the automation plan received from the server into the RPA tool and automates business procedures. The input is the automation plan. The output is the result of the executed automated tasks. Here, automated work is performed according to the plan through the RPA software.

[0309] Step 6:

[0310] The server monitors executed business procedures and collects performance data. The input is execution data obtained from the RPA tool. The output is a monitoring report containing metrics for business improvement. This provides a foundation for evaluating process efficiency and effectiveness and generating improvement proposals.

[0311] Step 7:

[0312] The user receives status reports and improvement suggestions for business processes from the server via a natural language interface. Input includes questions and confirmation requests from the user. Output provides the user with process status information and specific improvement suggestions. The use of a generative AI model enables natural and intuitive dialogue.

[0313] (Application Example 1)

[0314] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0315] Modern logistics facilities suffer from inefficiency and a high risk of human error due to the significant amount of manual work involved in managing the inbound and outbound movement of goods and arranging deliveries. Repetitive tasks, in particular, place a heavy burden on workers and contribute to overall operational efficiency. Solving this problem and improving efficiency within logistics facilities is therefore essential.

[0316] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0317] In this invention, the server includes means for collecting business data, analyzing that data to visualize business processes, identifying work procedures that can be automated from the visualized business processes, and generating an automation plan based on the identified work procedures. This enables efficient management of goods receiving and shipping and delivery arrangements within logistics facilities, reduces human labor, and allows for highly operationally efficient business processes.

[0318] "Business data" refers to information such as records of goods being received and shipped from logistics facilities, inventory information, and shipping instructions.

[0319] "Analysis" is the process of analyzing work patterns based on collected business data to highlight efficient workflows.

[0320] "Visualization" means presenting analysis results visually and providing them in a way that is easy for stakeholders to understand.

[0321] "Work procedure" refers to a series of specific operations or steps involved in performing a task.

[0322] An "automation plan diagram" is a drawing that describes the design for automating specific work procedures in an efficient and error-free manner.

[0323] "Automated equipment" is a general term for hardware and software used to automatically execute specific business procedures.

[0324] "Monitoring" is a process that tracks the status of ongoing work procedures and business processes in real time, enabling a quick response if problems occur.

[0325] An "improvement proposal" is a suggestion for changes or improvements to business processes based on collected monitoring data, aimed at improving their efficiency and accuracy.

[0326] "User" refers to the staff or managers of a logistics facility who use this system for their operations.

[0327] "Dialogue" is the process by which a user and a system interact and exchange information, either verbally or through text.

[0328] "Efficiency improvement" refers to making tasks possible with less time and resources.

[0329] The present invention provides an automation program aimed at improving the efficiency of operations in logistics facilities. The server uses APIs and database connections to collect operational data from various information sources within the logistics facility. This information includes inbound and outbound records, inventory information, and shipping instructions. This data is temporarily stored in a database and undergoes a pre-processing stage for analysis.

[0330] The server analyzes stored data using natural language processing (NLP) and machine learning models to visualize efficient business processes. For example, it automatically extracts the optimal work procedures for smooth receiving and shipping operations. Based on the work procedures identified through the analysis, an automation plan is generated, and the specific work flow is executed by an RPA tool. In this case, UiPath is used as the specific RPA tool.

[0331] The terminal monitors ongoing work procedures in real time and receives improvement suggestions from the server to enhance efficiency and accuracy. Users then adjust their work based on these suggestions. Oracle or MySQL may be used as the monitoring system on the terminal. Users can also check the status of business processes and inquire about improvements through a natural language conversational interface. This entire process is supported by cloud-based support on the AWS platform.

[0332] As a concrete example, when new home appliances arrive at a logistics facility, their barcodes are automatically scanned, and a database in the cloud is updated. Next, shipping instructions are automatically generated and sent to the delivery company. Workers can monitor this entire process via their smartphones and intervene with human judgment as needed.

[0333] An example of a prompt for a generating AI model is: "Please explain how to improve process efficiency using natural language processing and RPA in an automated shipping management system for a logistics center. Please include specific operating procedures and areas for improvement." This prompt allows the system to analyze the information and provide it to the user.

[0334] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0335] Step 1:

[0336] The server collects operational data from information systems within the logistics facility. As input, it retrieves data such as inbound / outbound records, inventory information, and shipping instructions using APIs and database connections. This data is temporarily stored in an SQL database to prepare it for subsequent processing.

[0337] Step 2:

[0338] The server analyzes stored data using natural language processing (NLP) techniques and machine learning models. The input is stored business data, and the output is visualization data representing efficient business processes. Specifically, pattern recognition is performed on each dataset to extract necessary features.

[0339] Step 3:

[0340] The server identifies automatable work procedures from the analyzed data and generates an automation plan. The input is visualized data, and the output is an automation plan that can be executed with an RPA tool. At this stage, it determines which procedures are best suited for automation based on the number of repetitions and frequency of each procedure.

[0341] Step 4:

[0342] The terminal imports the generated automation plan into the RPA tool (UiPath) and executes the automated work procedure. The input is the automation plan, and the output is the result of the performed work. Specifically, the automated process is initiated based on the plan, and the work procedure is carried out through a robotic process.

[0343] Step 5:

[0344] The system monitors the work procedures performed by the terminal in real time, and the server analyzes the collected monitoring data to generate improvement suggestions. The input is data recording the progress of the work, and the output is improvement suggestions for efficiency. In actual operation, system logs and execution performance are evaluated, and continuous optimization is performed.

[0345] Step 6:

[0346] Users utilize a natural language dialogue interface via their devices to understand the status of business processes and consider improvement proposals as needed. Input consists of user inquiries made through the dialogue interface, while output is information and suggestions related to the business. Specifically, users use smartphones or PCs to check the progress of their work and apply suggestions from the system.

[0347] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0348] This invention improves the quality of the user interface and supports more effective business efficiency by incorporating an emotion engine into business automation systems. This system optimizes business processes through interaction between servers, terminals, and users.

[0349] The server first collects business data from the organization's information systems and visualizes the business flow by analyzing that data. The business data analyzed by the server is processed to ensure that it is reflected in the business flow without any omissions. Then, it identifies processes that are suitable for automation from the visualized business flow. For the identified processes, the server generates an automation blueprint. This blueprint describes the business processing procedures, data flow, and error handling in detail, and can be imported into RPA tools.

[0350] The terminal uses this automation blueprint to execute business processes. The executed processes are monitored by the server, and performance data is used to generate improvement suggestions. This enables continuous improvement of business processes.

[0351] The emotion engine analyzes the user's emotional state in real time. The device uses the emotion engine to proactively recognize the user's emotions during communication and reflects the results in its service response. For example, if the device detects that the user is stressed, it will be configured to provide a more considerate response. Emotional data is also used to optimize the overall system's responsiveness and efficiency.

[0352] When users use the system, they can interact with servers and terminals through a natural language interface. At this time, insights based on the analysis results of the sentiment engine are provided, including improvement suggestions for user inquiries. This feature allows users to receive more emotionally resonant feedback on their work processes.

[0353] As a concrete example, in a customer support department that handles customer interactions, a system that detects user emotions can help operators determine the priority of their requests. Furthermore, for operators experiencing stress, the system can suggest automating processes to reduce their workload, thereby improving operational efficiency. This system enables increased productivity across the entire organization.

[0354] The following describes the processing flow.

[0355] Step 1:

[0356] The server collects business data from various information systems within the company. Using APIs and database connections, it efficiently imports data from systems such as email, CRM, and ERP, and stores it in an organized database.

[0357] Step 2:

[0358] The server analyzes collected business data and visualizes the business flow. It utilizes natural language processing technology and machine learning models to clearly define business processes and operation patterns, generating a visually represented model.

[0359] Step 3:

[0360] The server identifies processes suitable for automation from the visualized workflow. It selects processes with a high potential for efficiency improvement by considering specific criteria such as the frequency of repetitive tasks, the amount of manual operation, and the error rate.

[0361] Step 4:

[0362] The server generates an automation blueprint based on the identified business process. The blueprint includes procedures, data flow, and error handling, and is output in a format that can be imported into an RPA tool.

[0363] Step 5:

[0364] The terminal imports the blueprint generated by the server into the RPA tool and starts the automated execution of business processes. The configured bot performs a specific task and records the results in the database.

[0365] Step 6:

[0366] The server monitors the executed business processes and collects performance data. It monitors metrics such as execution speed, error rate, and success rate, and generates improvement suggestions based on this data.

[0367] Step 7:

[0368] The terminal communicates with the user through a natural language interface, providing process status reports and suggesting improvements. It also provides real-time answers to user questions and inquiries.

[0369] Step 8:

[0370] A device equipped with an emotion engine analyzes the user's emotional state. It extracts emotional data from communication with the user and uses it to improve service responses and suggest improvements, thereby enhancing the quality of the interaction.

[0371] Step 9:

[0372] Users can monitor the progress and results of business processes and discover opportunities for further business improvement through insights gained from the emotion engine. They can also propose new automated processes as needed.

[0373] (Example 2)

[0374] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0375] In today's business environment, there is a demand for both efficient business processes and user-friendliness. In particular, improving the quality of emotion-based interfaces during the process of automating business processes is a challenge. Furthermore, there is the difficulty of providing individually optimized services that meet the diverse needs of users.

[0376] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0377] In this invention, the server includes means for collecting business data, analyzing that data to visualize business processes, identifying tasks that can be automated from the visualized business processes, and generating automation designs based on the identified tasks. This improves the efficiency of operations and enables the provision of a flexible interface based on the user's emotional state.

[0378] "Business data" refers to information related to business processes obtained from information systems within an organization.

[0379] "Analysis" is the act of processing and analyzing collected data to extract useful information and patterns.

[0380] A "business process" refers to a series of procedures and activities necessary to perform a specific task.

[0381] "Visualization" is the act of representing data and information visually to make them easier to understand.

[0382] "Automated tasks" are those tasks performed by humans that can be made more efficient by applying automation technology.

[0383] "Automation design" is a detailed description of the procedures and methods for executing a process that can be automated.

[0384] A "process automation tool" is software used to automatically execute and manage business tasks.

[0385] "Monitoring" is the activity of continuously monitoring the operating status of a system or process and collecting data.

[0386] "Emotional analysis" is a technology that objectively analyzes a user's emotional state and represents that state using numerical values ​​or categories.

[0387] "Natural language dialogue" refers to communication between a system and a user using the language that humans use on a daily basis.

[0388] This invention is a system for streamlining business processes and improving user interaction. The main components of this system include servers, terminals, and users.

[0389] The server first collects business data from the organization's information systems. It uses database software to issue SQL queries and retrieve the necessary data. Next, it analyzes the data using Python libraries such as Pandas and NumPy to extract information for visualizing business processes. Based on the information obtained from this analysis, it creates flowcharts using visualization tools such as D3.js and Tableau.

[0390] The terminal performs automation design based on visualization data sent from the server. Specifically, it automates business tasks using process automation tools such as UiPath and Automation Anywhere. During the execution of automated tasks, it monitors and collects performance data using Prometheus. This data is visualized in a dashboard format using tools such as Grafana and used to improve business processes.

[0391] Furthermore, the device is equipped with emotion analysis capabilities, using image analysis and natural language processing technologies to analyze the user's emotions in real time. This process utilizes emotion analysis tools from Affectiva and IBM. By recognizing the user's emotional state, the device can provide more human-like and considerate responses.

[0392] Users interact with the system through a natural language interface. In response to user inquiries, the system provides appropriate feedback using prompts generated by a generative AI model. For example, if a user enters the request, "Please check the progress of the current task," the system analyzes real-time business data and reports its progress.

[0393] This system will improve the overall operational efficiency of the organization and enable it to provide users with services tailored to their individual needs.

[0394] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0395] Step 1:

[0396] The server collects business data from the organization's information systems. It uses raw business data obtained from databases as input. This data includes customer information, transaction history, inventory status, etc. SQL queries are executed to extract the data, which then becomes available as output within the server.

[0397] Step 2:

[0398] The server analyzes the collected business data to visualize business processes. The input is the business data collected in Step 1. This data is then statistically analyzed and cleansed using Python's Pandas and NumPy to gain new insights. Based on these insights, a business flowchart is created using visualization tools such as D3.js, and the visualized data is output.

[0399] Step 3:

[0400] The server identifies tasks that can be automated using visualization data. The input is the business flowchart generated in step 2. Through machine learning algorithms, it identifies repetitive tasks that are particularly likely to benefit from automation and outputs the results as an automation design.

[0401] Step 4:

[0402] The terminal automates business tasks based on automation designs provided by the server. It receives automation designs as input and uses UiPath to create automation scripts. For example, it might perform a task to automatically generate a standardized report at a fixed time each day, and output whether the task was completed on schedule.

[0403] Step 5:

[0404] The server monitors tasks performed on terminals and collects performance data. It uses logs and execution result data generated by UiPath as input. Prometheus is used to collect this data in real time, and Grafana is used to visualize it, yielding output that shows efficiency and changes in execution status.

[0405] Step 6:

[0406] The terminal performs emotion analysis during user interaction. Inputs include audio data and webcam footage from user conversations. Using technologies from Affectiva and IBM, it analyzes emotions, generates responses based on that data, and provides user-adapted services as output.

[0407] Step 7:

[0408] Users interact with the system using natural language and receive feedback. They input questions and commands in natural language using prompts generated by a generative AI model. The system analyzes these requests, generates responses based on monitoring and sentiment analysis data, and provides them to the user as output. For example, inputting a request such as "Please check the progress of the current task" will provide progress information.

[0409] (Application Example 2)

[0410] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0411] In modern information processing systems, improving operational efficiency and enhancing user experience are crucial challenges. In particular, there is a demand for services that respond to user emotions and individual needs. However, conventional systems lacked sufficient emotion analysis capabilities, making effective communication with users difficult. This resulted in increased workload and inconsistent service quality.

[0412] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0413] In this invention, the server includes means for collecting business data and analyzing that data to visualize the business flow; means for identifying business processes that can be automated from the visualized business flow; means for generating automation blueprints based on the identified business processes; means for importing the generated blueprints into an automation tool and executing the business processes; means for monitoring the executed business processes and generating improvement suggestions; and means for analyzing the user's emotional state and adjusting the response, and for providing individualized responses based on the user's emotional analysis. This enables efficient execution of business processes that take into account the user's emotional state and personalized user responses.

[0414] "Business data" refers to a collection of information and figures related to business processes generated within an organization.

[0415] A "business process flow" is a visual representation of how business processes progress and how data flows.

[0416] An "automation blueprint" is a design document that describes the specific steps and data flow required to automate an identified business process.

[0417] "Automation tools" is a general term for software or systems used to execute automation blueprints.

[0418] "Monitoring" is the act of observing and recording the operation and performance of a business process in progress.

[0419] An "improvement suggestion" is a specific proposal to improve the efficiency and quality of business processes based on monitoring data.

[0420] "Natural language communication" refers to dialogue and information exchange that takes place through the language that humans normally use.

[0421] "Emotional state" refers to the psychological state a user experiences emotionally, and includes changes in emotions that occur under specific circumstances.

[0422] "Individualized support" refers to providing specific support or services tailored to the individual needs and circumstances of each user.

[0423] This invention aims to improve the user experience by incorporating sentiment analysis functionality into a business automation system. The system operates between a server, a terminal, and a user, with each party's role clearly defined.

[0424] First, the server collects business data from the organization's information systems and stores it in a database. The stored business data is analyzed and visualized as a business flow. Data analysis tools (e.g., Tableau or Power BI) can be used as the software for this analysis. Visualization helps identify which business processes are suitable for automation and generates automation blueprints suitable for import into RPA (Robotic Process Automation) tools. These blueprints include process steps, data flow, and error handling.

[0425] The terminal executes business processes based on the generated blueprints. This process utilizes RPA tools installed on the terminal (e.g., UiPath or Automation Anywhere). During process execution, the server monitors its operation and collects performance data. Based on this data, the system generates process improvement suggestions.

[0426] Communication with users is conducted using natural language processing technologies (e.g., NLTK and spaCy). The terminal also uses an emotion analysis engine (e.g., Emotion API) to analyze the user's emotions in real time. If the user is experiencing stress, the system provides considerate responses from the terminal and offers suggestions to reduce their workload.

[0427] As a concrete example, in a system utilizing a home robot, the robot analyzes the user's emotions from their facial expressions and voice, and if it determines that the user is tired, it recommends relaxation music. An example of a prompt message to the generative AI model used in this case would be: "Program music suggestions for when the user is feeling tired on a Friday night. These suggestions should be based on emotion analysis data."

[0428] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0429] Step 1:

[0430] The server collects business data from the organization's information systems. The information obtained as input includes business process log data and error history. The server stores this data in a database, preparing it for subsequent analysis. The output is a well-organized business dataset.

[0431] Step 2:

[0432] The server visualizes the business flow based on the acquired data. It generates a flowchart from the input dataset and visualizes it using data analysis tools. Specifically, it analyzes the start time, end time, and duration of each business step. The output is a visual map showing the flow of the business process.

[0433] Step 3:

[0434] The server identifies business processes that can be automated from the visualized workflow. Machine learning algorithms are used for analysis, picking out repetitive sections within the process. A workflow map is used as input, and the identified automation candidate processes are output.

[0435] Step 4:

[0436] The server generates automation blueprints based on identified business processes. A template-based approach is used for blueprint generation. Detailed information about the identified processes is used as input data, and the output is a blueprint file for use in automation tools.

[0437] Step 5:

[0438] The terminal imports the automation blueprint into the RPA tool and executes the business process. The business is processed automatically according to the specified procedures. The input is the blueprint file, and the output is the automated business result.

[0439] Step 6:

[0440] The server monitors executed business processes and collects performance data. Monitoring tools are used to obtain data such as execution time and error rates. The input is the execution status of the business processes, and the output is a performance report.

[0441] Step 7:

[0442] The device uses an emotion analysis engine to analyze the user's emotional state in real time. It acquires the user's voice and video as input data and performs calculations to identify their emotional state. The output is the analysis result regarding the user's emotions.

[0443] Step 8:

[0444] The device provides personalized responses based on the user's sentiment analysis results. Based on the analysis, it generates appropriate responses and suggestions and presents them to the user. In this process, the sentiment analysis results are used as input, and the output is specific actions or suggestions.

[0445] Step 9:

[0446] Users interact with the system using natural language communication methods provided by their devices. Appropriate responses and improvement suggestions are provided based on user input. Input is the user's natural language, and output is relevant information and suggestions.

[0447] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0448] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0449] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0450] [Third Embodiment]

[0451] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0452] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0453] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0454] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0455] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0456] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0457] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0458] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0459] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0460] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0461] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0462] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0463] This invention provides a system for efficiently automating business processes. This system consists of multiple computer devices, with servers, terminals, and users working together. The following outlines its operation.

[0464] The server first collects business data from various information systems within the organization. During this process, the server continuously imports necessary data from systems such as email systems, CRM, ERP, and sales support systems using APIs and database connections. The collected data is temporarily stored in a database and becomes the subject of analysis.

[0465] Next, the server analyzes the stored business data. Natural language processing (NLP) techniques and machine learning models are used to automatically visualize the business flow from the data. For example, common data manipulation patterns by multiple employees can be extracted and modeled as typical business procedures.

[0466] Subsequently, the server identifies processes from the visualized workflow that are candidates for automation and can be handled using rule-based methods. This process evaluates priorities based on criteria such as the frequency of repetition of tasks, the complexity of manual work, and the error rate, and then determines which processes to automate.

[0467] For processes deemed suitable for automation, the server generates an automation blueprint. This blueprint records specific business procedures, data movement, and error handling methods, and is in a format that can be directly imported into RPA (Robotic Process Automation) tools.

[0468] Subsequently, the terminal imports the generated design into the RPA tool and begins automating the business process in the user's environment. After execution, the server monitors the process and generates improvement suggestions based on the collected performance data. These improvement suggestions aim to optimize and streamline the process and are used for communication with the user.

[0469] Furthermore, the terminal reports the status of business processes to the user via a natural language interface and provides improvement suggestions and answers to questions. In this way, users can interact with the system intuitively and effectively operate automated business processes.

[0470] As a concrete example, when automating the daily data aggregation process in the sales department, the server analyzes the data input patterns of each sales representative and generates an RPA blueprint to standardize the processing of unique data fields. The terminal then uses this blueprint to automatically generate aggregation reports, achieving efficient operations. This system dynamically optimizes the automation process, significantly reducing the burden on users.

[0471] The following describes the processing flow.

[0472] Step 1:

[0473] The server uses APIs and database connections to collect business data from various information systems within the organization, and automatically performs tasks such as periodically importing data from email systems, CRM, and ERP.

[0474] Step 2:

[0475] The server stores the collected business data in a database and analyzes the data using natural language processing and machine learning models. In this process, the business flow is automatically visualized from the data, and patterned procedures are modeled.

[0476] Step 3:

[0477] The server evaluates the business workflows identified from the analysis results and identifies processes that can be automated. Criteria for selecting processes include the frequency of repetition, the complexity of manual work, and the error rate.

[0478] Step 4:

[0479] The server generates an automation blueprint for the identified process. This blueprint includes specific business procedures, data movement methods, and error handling, and is in a format that can be imported into an RPA tool.

[0480] Step 5:

[0481] The terminal imports the blueprint generated by the server into the RPA tool and completes the necessary settings to start the automated business process. It then launches the bot in the execution environment.

[0482] Step 6:

[0483] Users can check the progress and results of ongoing business processes through their terminals and take timely action if problems arise. This ensures that the effectiveness of the processes in the real world is maintained.

[0484] Step 7:

[0485] The server continuously monitors performance data of executed business processes and records information on efficiency and failures. Based on this, it generates improvement suggestions for further optimization.

[0486] Step 8:

[0487] The terminal provides process status reports and answers user questions via a natural language interface, offering improvement suggestions. Through this interaction, users can gain concrete insights into improving process operations.

[0488] (Example 1)

[0489] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0490] In today's business environment, there is a demand for collecting necessary business information from vast amounts of information resources and analyzing it effectively. Furthermore, as business procedures are increasingly automated based on the analysis results, existing methods may not be sufficient. To address this challenge, it is necessary to develop a system that efficiently automates business procedures and effectively monitors and improves the execution process.

[0491] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0492] In this invention, the server includes means for collecting business information from information resources and analyzing that information to visualize business procedures; means for identifying business procedures that can be automated from the visualized business procedures; and means for incorporating the generated plan into automation technology and executing the business procedures. This makes it possible to efficiently advance the automation of business procedures and to easily monitor and improve them.

[0493] "Information resources" is a general term for multiple databases, applications, and systems that provide business information.

[0494] "Business information" refers to the data and knowledge necessary to perform or manage business operations within an organization.

[0495] "Analysis" is the process of processing collected business information and extracting meaningful patterns and trends.

[0496] "Business procedures" refer to a series of operations and steps necessary to efficiently carry out a business task.

[0497] "Visualization" refers to displaying extracted business procedures and data in a format that is easy for users to understand.

[0498] An "automated procedure" refers to a set of processes or operations designed to automate manual human work.

[0499] "Automation technology" refers to all technologies used to automate business procedures.

[0500] "Monitoring" refers to activities aimed at evaluating the performance of ongoing business procedures and processes and making improvements as needed.

[0501] A "proposal for improvement" refers to specific changes proposed to enhance the efficiency and effectiveness of business procedures and processes.

[0502] "Users" refer to those who operate the system or use the information obtained to perform their duties.

[0503] "Natural language dialogue" refers to providing an interface that uses everyday language so that users can intuitively communicate with the system.

[0504] The invention will now be described in terms of its implementation. This system works in which a server, terminals, and users collaborate to efficiently collect and analyze business information and automate business procedures.

[0505] The server first collects business information from information resources. To do this, the server uses APIs and database connections to retrieve business-related information from various management systems. Standard database management systems and interfaces such as REST APIs are used for information retrieval. The collected information is centrally stored in a database, preparing it for the next analysis step.

[0506] Next, the server analyzes the stored information using natural language processing techniques and machine learning models. Programming languages ​​such as Python and R, along with related libraries like TensorFlow and NLTK, are used. The analysis results are visually represented, providing the business procedures as a flow chart that is easily understandable to the user.

[0507] After completing the analysis and visualization, the server identifies business procedures that can be automated and generates an automation plan based on them. This plan describes the detailed steps of the business procedures and is in a format that can be imported into an RPA tool. The server creates the plan using specific RPA software (e.g., UiPath, Blue Prism).

[0508] The terminal automates business procedures by importing the generated plan into the RPA tool. The terminal reports the progress of the business procedures to the server, which monitors its performance and provides suggestions for improvement.

[0509] Users can interact with the system via a natural language interface and receive notifications about the status of automated tasks. They can also receive immediate answers to questions and requests. Generative AI models are used for system interaction, enabling natural and effective communication.

[0510] As a concrete example, consider data aggregation work in the sales department. This system analyzes the data entry patterns of each employee and generates an RPA plan for standardizing unique data fields. The terminal then uses this plan to automatically create aggregation reports, thereby improving operational efficiency.

[0511] An example of a prompt message would be, "Analyze the update patterns of the sales data and generate an RPA blueprint." By entering this prompt into the system, the server can begin the necessary analysis and blueprint generation.

[0512] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0513] Step 1:

[0514] The server collects business information from information resources. Using API keys and database connection information as input, it retrieves business-related data from email, customer relationship management systems, and corporate resource planning systems. The output is a database containing the raw data. This step converts the data into a consistent format according to a protocol, enabling efficient processing in subsequent analysis steps.

[0515] Step 2:

[0516] The server organizes the collected data and analyzes it using natural language processing (NLP) techniques. It uses the raw data obtained in the previous step as input, performing data processing such as noise reduction, tokenization, and summarization. The output is an analysis showing patterns and trends in business procedures. This analysis extracts significant information from the data using NLP libraries in Python or R.

[0517] Step 3:

[0518] The server visualizes business procedures based on the analysis results. The input is the analysis results from step 2. As output, flowcharts and dashboards are generated, presenting business procedures in a format that is easy for users to understand. As a tool, data visualization software is used to generate the visual information.

[0519] Step 4:

[0520] The server identifies parts of the business process that can be automated and creates an automation plan. The input is a visualized business process. The output is a plan containing detailed steps required for automation, in a format that can be imported into an RPA tool. This step uses algorithms to detect repetitive tasks and manual errors in the business flow and evaluate their priority.

[0521] Step 5:

[0522] The terminal imports the automation plan received from the server into the RPA tool and automates business procedures. The input is the automation plan. The output is the result of the executed automated tasks. Here, automated work is performed according to the plan through the RPA software.

[0523] Step 6:

[0524] The server monitors executed business procedures and collects performance data. The input is execution data obtained from the RPA tool. The output is a monitoring report containing metrics for business improvement. This provides a foundation for evaluating process efficiency and effectiveness and generating improvement proposals.

[0525] Step 7:

[0526] The user receives status reports and improvement suggestions for business processes from the server via a natural language interface. Input includes questions and confirmation requests from the user. Output provides the user with process status information and specific improvement suggestions. The use of a generative AI model enables natural and intuitive dialogue.

[0527] (Application Example 1)

[0528] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0529] Modern logistics facilities suffer from inefficiency and a high risk of human error due to the significant amount of manual work involved in managing the inbound and outbound movement of goods and arranging deliveries. Repetitive tasks, in particular, place a heavy burden on workers and contribute to overall operational efficiency. Solving this problem and improving efficiency within logistics facilities is therefore essential.

[0530] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0531] In this invention, the server includes means for collecting business data, analyzing that data to visualize business processes, identifying work procedures that can be automated from the visualized business processes, and generating an automation plan based on the identified work procedures. This enables efficient management of goods receiving and shipping and delivery arrangements within logistics facilities, reduces human labor, and allows for highly operationally efficient business processes.

[0532] "Business data" refers to information such as records of goods being received and shipped from logistics facilities, inventory information, and shipping instructions.

[0533] "Analysis" is the process of analyzing work patterns based on collected business data to highlight efficient workflows.

[0534] "Visualization" means presenting analysis results visually and providing them in a way that is easy for stakeholders to understand.

[0535] "Work procedure" refers to a series of specific operations or steps involved in performing a task.

[0536] An "automation plan diagram" is a drawing that describes the design for automating specific work procedures in an efficient and error-free manner.

[0537] "Automated equipment" is a general term for hardware and software used to automatically execute specific business procedures.

[0538] "Monitoring" is a process that tracks the status of ongoing work procedures and business processes in real time, enabling a quick response if problems occur.

[0539] An "improvement proposal" is a suggestion for changes or improvements to business processes based on collected monitoring data, aimed at improving their efficiency and accuracy.

[0540] "User" refers to the staff or managers of a logistics facility who use this system for their operations.

[0541] "Dialogue" is the process by which a user and a system interact and exchange information, either verbally or through text.

[0542] "Efficiency improvement" refers to making tasks possible with less time and resources.

[0543] The present invention provides an automation program aimed at improving the efficiency of operations in logistics facilities. The server uses APIs and database connections to collect operational data from various information sources within the logistics facility. This information includes inbound and outbound records, inventory information, and shipping instructions. This data is temporarily stored in a database and undergoes a pre-processing stage for analysis.

[0544] The server analyzes stored data using natural language processing (NLP) and machine learning models to visualize efficient business processes. For example, it automatically extracts the optimal work procedures for smooth receiving and shipping operations. Based on the work procedures identified through the analysis, an automation plan is generated, and the specific work flow is executed by an RPA tool. In this case, UiPath is used as the specific RPA tool.

[0545] The terminal monitors ongoing work procedures in real time and receives improvement suggestions from the server to enhance efficiency and accuracy. Users then adjust their work based on these suggestions. Oracle or MySQL may be used as the monitoring system on the terminal. Users can also check the status of business processes and inquire about improvements through a natural language conversational interface. This entire process is supported by cloud-based support on the AWS platform.

[0546] As a concrete example, when new home appliances arrive at a logistics facility, their barcodes are automatically scanned, and a database in the cloud is updated. Next, shipping instructions are automatically generated and sent to the delivery company. Workers can monitor this entire process via their smartphones and intervene with human judgment as needed.

[0547] An example of a prompt for a generating AI model is: "Please explain how to improve process efficiency using natural language processing and RPA in an automated shipping management system for a logistics center. Please include specific operating procedures and areas for improvement." This prompt allows the system to analyze the information and provide it to the user.

[0548] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0549] Step 1:

[0550] The server collects operational data from information systems within the logistics facility. As input, it retrieves data such as inbound / outbound records, inventory information, and shipping instructions using APIs and database connections. This data is temporarily stored in an SQL database to prepare it for subsequent processing.

[0551] Step 2:

[0552] The server analyzes stored data using natural language processing (NLP) techniques and machine learning models. The input is stored business data, and the output is visualization data representing efficient business processes. Specifically, pattern recognition is performed on each dataset to extract necessary features.

[0553] Step 3:

[0554] The server identifies automatable work procedures from the analyzed data and generates an automation plan. The input is visualized data, and the output is an automation plan that can be executed with an RPA tool. At this stage, it determines which procedures are best suited for automation based on the number of repetitions and frequency of each procedure.

[0555] Step 4:

[0556] The terminal imports the generated automation plan into the RPA tool (UiPath) and executes the automated work procedure. The input is the automation plan, and the output is the result of the performed work. Specifically, the automated process is initiated based on the plan, and the work procedure is carried out through a robotic process.

[0557] Step 5:

[0558] The system monitors the work procedures performed by the terminal in real time, and the server analyzes the collected monitoring data to generate improvement suggestions. The input is data recording the progress of the work, and the output is improvement suggestions for efficiency. In actual operation, system logs and execution performance are evaluated, and continuous optimization is performed.

[0559] Step 6:

[0560] Users utilize a natural language dialogue interface via their devices to understand the status of business processes and consider improvement proposals as needed. Input consists of user inquiries made through the dialogue interface, while output is information and suggestions related to the business. Specifically, users use smartphones or PCs to check the progress of their work and apply suggestions from the system.

[0561] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0562] This invention improves the quality of the user interface and supports more effective business efficiency by incorporating an emotion engine into business automation systems. This system optimizes business processes through interaction between servers, terminals, and users.

[0563] The server first collects business data from the organization's information systems and visualizes the business flow by analyzing that data. The business data analyzed by the server is processed to ensure that it is reflected in the business flow without any omissions. Then, it identifies processes that are suitable for automation from the visualized business flow. For the identified processes, the server generates an automation blueprint. This blueprint describes the business processing procedures, data flow, and error handling in detail, and can be imported into RPA tools.

[0564] The terminal uses this automation blueprint to execute business processes. The executed processes are monitored by the server, and performance data is used to generate improvement suggestions. This enables continuous improvement of business processes.

[0565] The emotion engine analyzes the user's emotional state in real time. The device uses the emotion engine to proactively recognize the user's emotions during communication and reflects the results in its service response. For example, if the device detects that the user is stressed, it will be configured to provide a more considerate response. Emotional data is also used to optimize the overall system's responsiveness and efficiency.

[0566] When users use the system, they can interact with servers and terminals through a natural language interface. At this time, insights based on the analysis results of the sentiment engine are provided, including improvement suggestions for user inquiries. This feature allows users to receive more emotionally resonant feedback on their work processes.

[0567] As a concrete example, in a customer support department that handles customer interactions, a system that detects user emotions can help operators determine the priority of their requests. Furthermore, for operators experiencing stress, the system can suggest automating processes to reduce their workload, thereby improving operational efficiency. This system enables increased productivity across the entire organization.

[0568] The following describes the processing flow.

[0569] Step 1:

[0570] The server collects business data from various information systems within the company. Using APIs and database connections, it efficiently imports data from systems such as email, CRM, and ERP, and stores it in an organized database.

[0571] Step 2:

[0572] The server analyzes collected business data and visualizes the business flow. It utilizes natural language processing technology and machine learning models to clearly define business processes and operation patterns, generating a visually represented model.

[0573] Step 3:

[0574] The server identifies processes suitable for automation from the visualized workflow. It selects processes with a high potential for efficiency improvement by considering specific criteria such as the frequency of repetitive tasks, the amount of manual operation, and the error rate.

[0575] Step 4:

[0576] The server generates an automation blueprint based on the identified business process. The blueprint includes procedures, data flow, and error handling, and is output in a format that can be imported into an RPA tool.

[0577] Step 5:

[0578] The terminal imports the blueprint generated by the server into the RPA tool and starts the automated execution of business processes. The configured bot performs a specific task and records the results in the database.

[0579] Step 6:

[0580] The server monitors the executed business processes and collects performance data. It monitors metrics such as execution speed, error rate, and success rate, and generates improvement suggestions based on this data.

[0581] Step 7:

[0582] The terminal communicates with the user through a natural language interface, providing process status reports and suggesting improvements. It also provides real-time answers to user questions and inquiries.

[0583] Step 8:

[0584] A device equipped with an emotion engine analyzes the user's emotional state. It extracts emotional data from communication with the user and uses it to improve service responses and suggest improvements, thereby enhancing the quality of the interaction.

[0585] Step 9:

[0586] Users can monitor the progress and results of business processes and discover opportunities for further business improvement through insights gained from the emotion engine. They can also propose new automated processes as needed.

[0587] (Example 2)

[0588] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0589] In today's business environment, there is a demand for both efficient business processes and user-friendliness. In particular, improving the quality of emotion-based interfaces during the process of automating business processes is a challenge. Furthermore, there is the difficulty of providing individually optimized services that meet the diverse needs of users.

[0590] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0591] In this invention, the server includes means for collecting business data, analyzing that data to visualize business processes, identifying tasks that can be automated from the visualized business processes, and generating automation designs based on the identified tasks. This improves the efficiency of operations and enables the provision of a flexible interface based on the user's emotional state.

[0592] "Business data" refers to information related to business processes obtained from information systems within an organization.

[0593] "Analysis" is the act of processing and analyzing collected data to extract useful information and patterns.

[0594] A "business process" refers to a series of procedures and activities necessary to perform a specific task.

[0595] "Visualization" is the act of representing data and information visually to make them easier to understand.

[0596] "Automated tasks" are those tasks performed by humans that can be made more efficient by applying automation technology.

[0597] "Automation design" is a detailed description of the procedures and methods for executing a process that can be automated.

[0598] A "process automation tool" is software used to automatically execute and manage business tasks.

[0599] "Monitoring" is the activity of continuously monitoring the operating status of a system or process and collecting data.

[0600] "Emotional analysis" is a technology that objectively analyzes a user's emotional state and represents that state using numerical values ​​or categories.

[0601] "Natural language dialogue" refers to communication between a system and a user using the language that humans use on a daily basis.

[0602] This invention is a system for streamlining business processes and improving user interaction. The main components of this system include servers, terminals, and users.

[0603] The server first collects business data from the organization's information systems. It uses database software to issue SQL queries and retrieve the necessary data. Next, it analyzes the data using Python libraries such as Pandas and NumPy to extract information for visualizing business processes. Based on the information obtained from this analysis, it creates flowcharts using visualization tools such as D3.js and Tableau.

[0604] The terminal performs automation design based on visualization data sent from the server. Specifically, it automates business tasks using process automation tools such as UiPath and Automation Anywhere. During the execution of automated tasks, it monitors and collects performance data using Prometheus. This data is visualized in a dashboard format using tools such as Grafana and used to improve business processes.

[0605] Furthermore, the device is equipped with emotion analysis capabilities, using image analysis and natural language processing technologies to analyze the user's emotions in real time. This process utilizes emotion analysis tools from Affectiva and IBM. By recognizing the user's emotional state, the device can provide more human-like and considerate responses.

[0606] Users interact with the system through a natural language interface. In response to user inquiries, the system provides appropriate feedback using prompts generated by a generative AI model. For example, if a user enters the request, "Please check the progress of the current task," the system analyzes real-time business data and reports its progress.

[0607] This system will improve the overall operational efficiency of the organization and enable it to provide users with services tailored to their individual needs.

[0608] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0609] Step 1:

[0610] The server collects business data from the organization's information systems. It uses raw business data obtained from databases as input. This data includes customer information, transaction history, inventory status, etc. SQL queries are executed to extract the data, which then becomes available as output within the server.

[0611] Step 2:

[0612] The server analyzes the collected business data to visualize business processes. The input is the business data collected in Step 1. This data is then statistically analyzed and cleansed using Python's Pandas and NumPy to gain new insights. Based on these insights, a business flowchart is created using visualization tools such as D3.js, and the visualized data is output.

[0613] Step 3:

[0614] The server identifies tasks that can be automated using visualization data. The input is the business flowchart generated in step 2. Through machine learning algorithms, it identifies repetitive tasks that are particularly likely to benefit from automation and outputs the results as an automation design.

[0615] Step 4:

[0616] The terminal automates business tasks based on automation designs provided by the server. It receives automation designs as input and uses UiPath to create automation scripts. For example, it might perform a task to automatically generate a standardized report at a fixed time each day, and output whether the task was completed on schedule.

[0617] Step 5:

[0618] The server monitors tasks performed on terminals and collects performance data. It uses logs and execution result data generated by UiPath as input. Prometheus is used to collect this data in real time, and Grafana is used to visualize it, yielding output that shows efficiency and changes in execution status.

[0619] Step 6:

[0620] The terminal performs emotion analysis during user interaction. Inputs include audio data and webcam footage from user conversations. Using technologies from Affectiva and IBM, it analyzes emotions, generates responses based on that data, and provides user-adapted services as output.

[0621] Step 7:

[0622] Users interact with the system using natural language and receive feedback. They input questions and commands in natural language using prompts generated by a generative AI model. The system analyzes these requests, generates responses based on monitoring and sentiment analysis data, and provides them to the user as output. For example, inputting a request such as "Please check the progress of the current task" will provide progress information.

[0623] (Application Example 2)

[0624] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0625] In modern information processing systems, improving operational efficiency and enhancing user experience are crucial challenges. In particular, there is a demand for services that respond to user emotions and individual needs. However, conventional systems lacked sufficient emotion analysis capabilities, making effective communication with users difficult. This resulted in increased workload and inconsistent service quality.

[0626] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0627] In this invention, the server includes means for collecting business data and analyzing that data to visualize the business flow; means for identifying business processes that can be automated from the visualized business flow; means for generating automation blueprints based on the identified business processes; means for importing the generated blueprints into an automation tool and executing the business processes; means for monitoring the executed business processes and generating improvement suggestions; and means for analyzing the user's emotional state and adjusting the response, and for providing individualized responses based on the user's emotional analysis. This enables efficient execution of business processes that take into account the user's emotional state and personalized user responses.

[0628] "Business data" refers to a collection of information and figures related to business processes generated within an organization.

[0629] A "business process flow" is a visual representation of how business processes progress and how data flows.

[0630] An "automation blueprint" is a design document that describes the specific steps and data flow required to automate an identified business process.

[0631] "Automation tools" is a general term for software or systems used to execute automation blueprints.

[0632] "Monitoring" is the act of observing and recording the operation and performance of a business process in progress.

[0633] An "improvement suggestion" is a specific proposal to improve the efficiency and quality of business processes based on monitoring data.

[0634] "Natural language communication" refers to dialogue and information exchange that takes place through the language that humans normally use.

[0635] "Emotional state" refers to the psychological state a user experiences emotionally, and includes changes in emotions that occur under specific circumstances.

[0636] "Individualized support" refers to providing specific support or services tailored to the individual needs and circumstances of each user.

[0637] This invention aims to improve the user experience by incorporating sentiment analysis functionality into a business automation system. The system operates between a server, a terminal, and a user, with each party's role clearly defined.

[0638] First, the server collects business data from the organization's information systems and stores it in a database. The stored business data is analyzed and visualized as a business flow. Data analysis tools (e.g., Tableau or Power BI) can be used as the software for this analysis. Visualization helps identify which business processes are suitable for automation and generates automation blueprints suitable for import into RPA (Robotic Process Automation) tools. These blueprints include process steps, data flow, and error handling.

[0639] The terminal executes business processes based on the generated blueprints. This process utilizes RPA tools installed on the terminal (e.g., UiPath or Automation Anywhere). During process execution, the server monitors its operation and collects performance data. Based on this data, the system generates process improvement suggestions.

[0640] Communication with users is conducted using natural language processing technologies (e.g., NLTK and spaCy). The terminal also uses an emotion analysis engine (e.g., Emotion API) to analyze the user's emotions in real time. If the user is experiencing stress, the system provides considerate responses from the terminal and offers suggestions to reduce their workload.

[0641] As a concrete example, in a system utilizing a home robot, the robot analyzes the user's emotions from their facial expressions and voice, and if it determines that the user is tired, it recommends relaxation music. An example of a prompt message to the generative AI model used in this case would be: "Program music suggestions for when the user is feeling tired on a Friday night. These suggestions should be based on emotion analysis data."

[0642] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0643] Step 1:

[0644] The server collects business data from the organization's information systems. The information obtained as input includes business process log data and error history. The server stores this data in a database, preparing it for subsequent analysis. The output is a well-organized business dataset.

[0645] Step 2:

[0646] The server visualizes the business flow based on the acquired data. It generates a flowchart from the input dataset and visualizes it using data analysis tools. Specifically, it analyzes the start time, end time, and duration of each business step. The output is a visual map showing the flow of the business process.

[0647] Step 3:

[0648] The server identifies business processes that can be automated from the visualized workflow. Machine learning algorithms are used for analysis, picking out repetitive sections within the process. A workflow map is used as input, and the identified automation candidate processes are output.

[0649] Step 4:

[0650] The server generates automation blueprints based on identified business processes. A template-based approach is used for blueprint generation. Detailed information about the identified processes is used as input data, and the output is a blueprint file for use in automation tools.

[0651] Step 5:

[0652] The terminal imports the automation blueprint into the RPA tool and executes the business process. The business is processed automatically according to the specified procedures. The input is the blueprint file, and the output is the automated business result.

[0653] Step 6:

[0654] The server monitors executed business processes and collects performance data. Monitoring tools are used to obtain data such as execution time and error rates. The input is the execution status of the business processes, and the output is a performance report.

[0655] Step 7:

[0656] The device uses an emotion analysis engine to analyze the user's emotional state in real time. It acquires the user's voice and video as input data and performs calculations to identify their emotional state. The output is the analysis result regarding the user's emotions.

[0657] Step 8:

[0658] The device provides personalized responses based on the user's sentiment analysis results. Based on the analysis, it generates appropriate responses and suggestions and presents them to the user. In this process, the sentiment analysis results are used as input, and the output is specific actions or suggestions.

[0659] Step 9:

[0660] Users interact with the system using natural language communication methods provided by their devices. Appropriate responses and improvement suggestions are provided based on user input. Input is the user's natural language, and output is relevant information and suggestions.

[0661] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0662] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0663] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0664] [Fourth Embodiment]

[0665] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0666] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0667] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0668] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0669] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0670] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0671] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0672] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0673] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0674] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0675] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0676] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0677] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0678] This invention provides a system for efficiently automating business processes. This system consists of multiple computer devices, with servers, terminals, and users working together. The following outlines its operation.

[0679] The server first collects business data from various information systems within the organization. During this process, the server continuously imports necessary data from systems such as email systems, CRM, ERP, and sales support systems using APIs and database connections. The collected data is temporarily stored in a database and becomes the subject of analysis.

[0680] Next, the server analyzes the stored business data. Natural language processing (NLP) techniques and machine learning models are used to automatically visualize the business flow from the data. For example, common data manipulation patterns by multiple employees can be extracted and modeled as typical business procedures.

[0681] Subsequently, the server identifies processes from the visualized workflow that are candidates for automation and can be handled using rule-based methods. This process evaluates priorities based on criteria such as the frequency of repetition of tasks, the complexity of manual work, and the error rate, and then determines which processes to automate.

[0682] For processes deemed suitable for automation, the server generates an automation blueprint. This blueprint records specific business procedures, data movement, and error handling methods, and is in a format that can be directly imported into RPA (Robotic Process Automation) tools.

[0683] Subsequently, the terminal imports the generated design into the RPA tool and begins automating the business process in the user's environment. After execution, the server monitors the process and generates improvement suggestions based on the collected performance data. These improvement suggestions aim to optimize and streamline the process and are used for communication with the user.

[0684] Furthermore, the terminal reports the status of business processes to the user via a natural language interface and provides improvement suggestions and answers to questions. In this way, users can interact with the system intuitively and effectively operate automated business processes.

[0685] As a concrete example, when automating the daily data aggregation process in the sales department, the server analyzes the data input patterns of each sales representative and generates an RPA blueprint to standardize the processing of unique data fields. The terminal then uses this blueprint to automatically generate aggregation reports, achieving efficient operations. This system dynamically optimizes the automation process, significantly reducing the burden on users.

[0686] The following describes the processing flow.

[0687] Step 1:

[0688] The server uses APIs and database connections to collect business data from various information systems within the organization, and automatically performs tasks such as periodically importing data from email systems, CRM, and ERP.

[0689] Step 2:

[0690] The server stores the collected business data in a database and analyzes the data using natural language processing and machine learning models. In this process, the business flow is automatically visualized from the data, and patterned procedures are modeled.

[0691] Step 3:

[0692] The server evaluates the business workflows identified from the analysis results and identifies processes that can be automated. Criteria for selecting processes include the frequency of repetition, the complexity of manual work, and the error rate.

[0693] Step 4:

[0694] The server generates an automation blueprint for the identified process. This blueprint includes specific business procedures, data movement methods, and error handling, and is in a format that can be imported into an RPA tool.

[0695] Step 5:

[0696] The terminal imports the blueprint generated by the server into the RPA tool and completes the necessary settings to start the automated business process. It then launches the bot in the execution environment.

[0697] Step 6:

[0698] Users can check the progress and results of ongoing business processes through their terminals and take timely action if problems arise. This ensures that the effectiveness of the processes in the real world is maintained.

[0699] Step 7:

[0700] The server continuously monitors performance data of executed business processes and records information on efficiency and failures. Based on this, it generates improvement suggestions for further optimization.

[0701] Step 8:

[0702] The terminal provides process status reports and answers user questions via a natural language interface, offering improvement suggestions. Through this interaction, users can gain concrete insights into improving process operations.

[0703] (Example 1)

[0704] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0705] In today's business environment, there is a demand for collecting necessary business information from vast amounts of information resources and analyzing it effectively. Furthermore, as business procedures are increasingly automated based on the analysis results, existing methods may not be sufficient. To address this challenge, it is necessary to develop a system that efficiently automates business procedures and effectively monitors and improves the execution process.

[0706] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0707] In this invention, the server includes means for collecting business information from information resources and analyzing that information to visualize business procedures; means for identifying business procedures that can be automated from the visualized business procedures; and means for incorporating the generated plan into automation technology and executing the business procedures. This makes it possible to efficiently advance the automation of business procedures and to easily monitor and improve them.

[0708] "Information resources" is a general term for multiple databases, applications, and systems that provide business information.

[0709] "Business information" refers to the data and knowledge necessary to perform or manage business operations within an organization.

[0710] "Analysis" is the process of processing collected business information and extracting meaningful patterns and trends.

[0711] "Business procedures" refer to a series of operations and steps necessary to efficiently carry out a business task.

[0712] "Visualization" refers to displaying extracted business procedures and data in a format that is easy for users to understand.

[0713] An "automated procedure" refers to a set of processes or operations designed to automate manual human work.

[0714] "Automation technology" refers to all technologies used to automate business procedures.

[0715] "Monitoring" refers to activities aimed at evaluating the performance of ongoing business procedures and processes and making improvements as needed.

[0716] A "proposal for improvement" refers to specific changes proposed to enhance the efficiency and effectiveness of business procedures and processes.

[0717] "Users" refer to those who operate the system or use the information obtained to perform their duties.

[0718] "Natural language dialogue" refers to providing an interface that uses everyday language so that users can intuitively communicate with the system.

[0719] The invention will now be described in terms of its implementation. This system works in which a server, terminals, and users collaborate to efficiently collect and analyze business information and automate business procedures.

[0720] The server first collects business information from information resources. To do this, the server uses APIs and database connections to retrieve business-related information from various management systems. Standard database management systems and interfaces such as REST APIs are used for information retrieval. The collected information is centrally stored in a database, preparing it for the next analysis step.

[0721] Next, the server analyzes the stored information using natural language processing techniques and machine learning models. Programming languages ​​such as Python and R, along with related libraries like TensorFlow and NLTK, are used. The analysis results are visually represented, providing the business procedures as a flow chart that is easily understandable to the user.

[0722] After completing the analysis and visualization, the server identifies business procedures that can be automated and generates an automation plan based on them. This plan describes the detailed steps of the business procedures and is in a format that can be imported into an RPA tool. The server creates the plan using specific RPA software (e.g., UiPath, Blue Prism).

[0723] The terminal automates business procedures by importing the generated plan into the RPA tool. The terminal reports the progress of the business procedures to the server, which monitors its performance and provides suggestions for improvement.

[0724] Users can interact with the system via a natural language interface and receive notifications about the status of automated tasks. They can also receive immediate answers to questions and requests. Generative AI models are used for system interaction, enabling natural and effective communication.

[0725] As a concrete example, consider data aggregation work in the sales department. This system analyzes the data entry patterns of each employee and generates an RPA plan for standardizing unique data fields. The terminal then uses this plan to automatically create aggregation reports, thereby improving operational efficiency.

[0726] An example of a prompt message would be, "Analyze the update patterns of the sales data and generate an RPA blueprint." By entering this prompt into the system, the server can begin the necessary analysis and blueprint generation.

[0727] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0728] Step 1:

[0729] The server collects business information from information resources. Using API keys and database connection information as input, it retrieves business-related data from email, customer relationship management systems, and corporate resource planning systems. The output is a database containing the raw data. This step converts the data into a consistent format according to a protocol, enabling efficient processing in subsequent analysis steps.

[0730] Step 2:

[0731] The server organizes the collected data and analyzes it using natural language processing (NLP) techniques. It uses the raw data obtained in the previous step as input, performing data processing such as noise reduction, tokenization, and summarization. The output is an analysis showing patterns and trends in business procedures. This analysis extracts significant information from the data using NLP libraries in Python or R.

[0732] Step 3:

[0733] The server visualizes business procedures based on the analysis results. The input is the analysis results from step 2. As output, flowcharts and dashboards are generated, presenting business procedures in a format that is easy for users to understand. As a tool, data visualization software is used to generate the visual information.

[0734] Step 4:

[0735] The server identifies parts of the business process that can be automated and creates an automation plan. The input is a visualized business process. The output is a plan containing detailed steps required for automation, in a format that can be imported into an RPA tool. This step uses algorithms to detect repetitive tasks and manual errors in the business flow and evaluate their priority.

[0736] Step 5:

[0737] The terminal imports the automation plan received from the server into the RPA tool and automates business procedures. The input is the automation plan. The output is the result of the executed automated tasks. Here, automated work is performed according to the plan through the RPA software.

[0738] Step 6:

[0739] The server monitors executed business procedures and collects performance data. The input is execution data obtained from the RPA tool. The output is a monitoring report containing metrics for business improvement. This provides a foundation for evaluating process efficiency and effectiveness and generating improvement proposals.

[0740] Step 7:

[0741] The user receives status reports and improvement suggestions for business processes from the server via a natural language interface. Input includes questions and confirmation requests from the user. Output provides the user with process status information and specific improvement suggestions. The use of a generative AI model enables natural and intuitive dialogue.

[0742] (Application Example 1)

[0743] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0744] Modern logistics facilities suffer from inefficiency and a high risk of human error due to the significant amount of manual work involved in managing the inbound and outbound movement of goods and arranging deliveries. Repetitive tasks, in particular, place a heavy burden on workers and contribute to overall operational efficiency. Solving this problem and improving efficiency within logistics facilities is therefore essential.

[0745] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0746] In this invention, the server includes means for collecting business data, analyzing that data to visualize business processes, identifying work procedures that can be automated from the visualized business processes, and generating an automation plan based on the identified work procedures. This enables efficient management of goods receiving and shipping and delivery arrangements within logistics facilities, reduces human labor, and allows for highly operationally efficient business processes.

[0747] "Business data" refers to information such as records of goods being received and shipped from logistics facilities, inventory information, and shipping instructions.

[0748] "Analysis" is the process of analyzing work patterns based on collected business data to highlight efficient workflows.

[0749] "Visualization" means presenting analysis results visually and providing them in a way that is easy for stakeholders to understand.

[0750] "Work procedure" refers to a series of specific operations or steps involved in performing a task.

[0751] An "automation plan diagram" is a drawing that describes the design for automating specific work procedures in an efficient and error-free manner.

[0752] "Automated equipment" is a general term for hardware and software used to automatically execute specific business procedures.

[0753] "Monitoring" is a process that tracks the status of ongoing work procedures and business processes in real time, enabling a quick response if problems occur.

[0754] An "improvement proposal" is a suggestion for changes or improvements to business processes based on collected monitoring data, aimed at improving their efficiency and accuracy.

[0755] "User" refers to the staff or managers of a logistics facility who use this system for their operations.

[0756] "Dialogue" is the process by which a user and a system interact and exchange information, either verbally or through text.

[0757] "Efficiency improvement" refers to making tasks possible with less time and resources.

[0758] The present invention provides an automation program aimed at improving the efficiency of operations in logistics facilities. The server uses APIs and database connections to collect operational data from various information sources within the logistics facility. This information includes inbound and outbound records, inventory information, and shipping instructions. This data is temporarily stored in a database and undergoes a pre-processing stage for analysis.

[0759] The server analyzes stored data using natural language processing (NLP) and machine learning models to visualize efficient business processes. For example, it automatically extracts the optimal work procedures for smooth receiving and shipping operations. Based on the work procedures identified through the analysis, an automation plan is generated, and the specific work flow is executed by an RPA tool. In this case, UiPath is used as the specific RPA tool.

[0760] The terminal monitors ongoing work procedures in real time and receives improvement suggestions from the server to enhance efficiency and accuracy. Users then adjust their work based on these suggestions. Oracle or MySQL may be used as the monitoring system on the terminal. Users can also check the status of business processes and inquire about improvements through a natural language conversational interface. This entire process is supported by cloud-based support on the AWS platform.

[0761] As a concrete example, when new home appliances arrive at a logistics facility, their barcodes are automatically scanned, and a database in the cloud is updated. Next, shipping instructions are automatically generated and sent to the delivery company. Workers can monitor this entire process via their smartphones and intervene with human judgment as needed.

[0762] An example of a prompt for a generating AI model is: "Please explain how to improve process efficiency using natural language processing and RPA in an automated shipping management system for a logistics center. Please include specific operating procedures and areas for improvement." This prompt allows the system to analyze the information and provide it to the user.

[0763] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0764] Step 1:

[0765] The server collects operational data from information systems within the logistics facility. As input, it retrieves data such as inbound / outbound records, inventory information, and shipping instructions using APIs and database connections. This data is temporarily stored in an SQL database to prepare it for subsequent processing.

[0766] Step 2:

[0767] The server analyzes stored data using natural language processing (NLP) techniques and machine learning models. The input is stored business data, and the output is visualization data representing efficient business processes. Specifically, pattern recognition is performed on each dataset to extract necessary features.

[0768] Step 3:

[0769] The server identifies automatable work procedures from the analyzed data and generates an automation plan. The input is visualized data, and the output is an automation plan that can be executed with an RPA tool. At this stage, it determines which procedures are best suited for automation based on the number of repetitions and frequency of each procedure.

[0770] Step 4:

[0771] The terminal imports the generated automation plan into the RPA tool (UiPath) and executes the automated work procedure. The input is the automation plan, and the output is the result of the performed work. Specifically, the automated process is initiated based on the plan, and the work procedure is carried out through a robotic process.

[0772] Step 5:

[0773] The system monitors the work procedures performed by the terminal in real time, and the server analyzes the collected monitoring data to generate improvement suggestions. The input is data recording the progress of the work, and the output is improvement suggestions for efficiency. In actual operation, system logs and execution performance are evaluated, and continuous optimization is performed.

[0774] Step 6:

[0775] Users utilize a natural language dialogue interface via their devices to understand the status of business processes and consider improvement proposals as needed. Input consists of user inquiries made through the dialogue interface, while output is information and suggestions related to the business. Specifically, users use smartphones or PCs to check the progress of their work and apply suggestions from the system.

[0776] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0777] This invention improves the quality of the user interface and supports more effective business efficiency by incorporating an emotion engine into business automation systems. This system optimizes business processes through interaction between servers, terminals, and users.

[0778] The server first collects business data from the organization's information systems and visualizes the business flow by analyzing that data. The business data analyzed by the server is processed to ensure that it is reflected in the business flow without any omissions. Then, it identifies processes that are suitable for automation from the visualized business flow. For the identified processes, the server generates an automation blueprint. This blueprint describes the business processing procedures, data flow, and error handling in detail, and can be imported into RPA tools.

[0779] The terminal uses this automation blueprint to execute business processes. The executed processes are monitored by the server, and performance data is used to generate improvement suggestions. This enables continuous improvement of business processes.

[0780] The emotion engine analyzes the user's emotional state in real time. The device uses the emotion engine to proactively recognize the user's emotions during communication and reflects the results in its service response. For example, if the device detects that the user is stressed, it will be configured to provide a more considerate response. Emotional data is also used to optimize the overall system's responsiveness and efficiency.

[0781] When users use the system, they can interact with servers and terminals through a natural language interface. At this time, insights based on the analysis results of the sentiment engine are provided, including improvement suggestions for user inquiries. This feature allows users to receive more emotionally resonant feedback on their work processes.

[0782] As a concrete example, in a customer support department that handles customer interactions, a system that detects user emotions can help operators determine the priority of their requests. Furthermore, for operators experiencing stress, the system can suggest automating processes to reduce their workload, thereby improving operational efficiency. This system enables increased productivity across the entire organization.

[0783] The following describes the processing flow.

[0784] Step 1:

[0785] The server collects business data from various information systems within the company. Using APIs and database connections, it efficiently imports data from systems such as email, CRM, and ERP, and stores it in an organized database.

[0786] Step 2:

[0787] The server analyzes collected business data and visualizes the business flow. It utilizes natural language processing technology and machine learning models to clearly define business processes and operation patterns, generating a visually represented model.

[0788] Step 3:

[0789] The server identifies processes suitable for automation from the visualized workflow. It selects processes with a high potential for efficiency improvement by considering specific criteria such as the frequency of repetitive tasks, the amount of manual operation, and the error rate.

[0790] Step 4:

[0791] The server generates an automation blueprint based on the identified business process. The blueprint includes procedures, data flow, and error handling, and is output in a format that can be imported into an RPA tool.

[0792] Step 5:

[0793] The terminal imports the blueprint generated by the server into the RPA tool and starts the automated execution of business processes. The configured bot performs a specific task and records the results in the database.

[0794] Step 6:

[0795] The server monitors the executed business processes and collects performance data. It monitors metrics such as execution speed, error rate, and success rate, and generates improvement suggestions based on this data.

[0796] Step 7:

[0797] The terminal communicates with the user through a natural language interface, providing process status reports and suggesting improvements. It also provides real-time answers to user questions and inquiries.

[0798] Step 8:

[0799] A device equipped with an emotion engine analyzes the user's emotional state. It extracts emotional data from communication with the user and uses it to improve service responses and suggest improvements, thereby enhancing the quality of the interaction.

[0800] Step 9:

[0801] Users can monitor the progress and results of business processes and discover opportunities for further business improvement through insights gained from the emotion engine. They can also propose new automated processes as needed.

[0802] (Example 2)

[0803] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0804] In today's business environment, there is a demand for both efficient business processes and user-friendliness. In particular, improving the quality of emotion-based interfaces during the process of automating business processes is a challenge. Furthermore, there is the difficulty of providing individually optimized services that meet the diverse needs of users.

[0805] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0806] In this invention, the server includes means for collecting business data, analyzing that data to visualize business processes, identifying tasks that can be automated from the visualized business processes, and generating automation designs based on the identified tasks. This improves the efficiency of operations and enables the provision of a flexible interface based on the user's emotional state.

[0807] "Business data" refers to information related to business processes obtained from information systems within an organization.

[0808] "Analysis" is the act of processing and analyzing collected data to extract useful information and patterns.

[0809] A "business process" refers to a series of procedures and activities necessary to perform a specific task.

[0810] "Visualization" is the act of representing data and information visually to make them easier to understand.

[0811] "Automated tasks" are those tasks performed by humans that can be made more efficient by applying automation technology.

[0812] "Automation design" is a detailed description of the procedures and methods for executing a process that can be automated.

[0813] A "process automation tool" is software used to automatically execute and manage business tasks.

[0814] "Monitoring" is the activity of continuously monitoring the operating status of a system or process and collecting data.

[0815] "Emotional analysis" is a technology that objectively analyzes a user's emotional state and represents that state using numerical values ​​or categories.

[0816] "Natural language dialogue" refers to communication between a system and a user using the language that humans use on a daily basis.

[0817] This invention is a system for streamlining business processes and improving user interaction. The main components of this system include servers, terminals, and users.

[0818] The server first collects business data from the organization's information systems. It uses database software to issue SQL queries and retrieve the necessary data. Next, it analyzes the data using Python libraries such as Pandas and NumPy to extract information for visualizing business processes. Based on the information obtained from this analysis, it creates flowcharts using visualization tools such as D3.js and Tableau.

[0819] The terminal performs automation design based on visualization data sent from the server. Specifically, it automates business tasks using process automation tools such as UiPath and Automation Anywhere. During the execution of automated tasks, it monitors and collects performance data using Prometheus. This data is visualized in a dashboard format using tools such as Grafana and used to improve business processes.

[0820] Furthermore, the device is equipped with emotion analysis capabilities, using image analysis and natural language processing technologies to analyze the user's emotions in real time. This process utilizes emotion analysis tools from Affectiva and IBM. By recognizing the user's emotional state, the device can provide more human-like and considerate responses.

[0821] Users interact with the system through a natural language interface. In response to user inquiries, the system provides appropriate feedback using prompts generated by a generative AI model. For example, if a user enters the request, "Please check the progress of the current task," the system analyzes real-time business data and reports its progress.

[0822] This system will improve the overall operational efficiency of the organization and enable it to provide users with services tailored to their individual needs.

[0823] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0824] Step 1:

[0825] The server collects business data from the organization's information systems. It uses raw business data obtained from databases as input. This data includes customer information, transaction history, inventory status, etc. SQL queries are executed to extract the data, which then becomes available as output within the server.

[0826] Step 2:

[0827] The server analyzes the collected business data to visualize business processes. The input is the business data collected in Step 1. This data is then statistically analyzed and cleansed using Python's Pandas and NumPy to gain new insights. Based on these insights, a business flowchart is created using visualization tools such as D3.js, and the visualized data is output.

[0828] Step 3:

[0829] The server identifies tasks that can be automated using visualization data. The input is the business flowchart generated in step 2. Through machine learning algorithms, it identifies repetitive tasks that are particularly likely to benefit from automation and outputs the results as an automation design.

[0830] Step 4:

[0831] The terminal automates business tasks based on automation designs provided by the server. It receives automation designs as input and uses UiPath to create automation scripts. For example, it might perform a task to automatically generate a standardized report at a fixed time each day, and output whether the task was completed on schedule.

[0832] Step 5:

[0833] The server monitors tasks performed on terminals and collects performance data. It uses logs and execution result data generated by UiPath as input. Prometheus is used to collect this data in real time, and Grafana is used to visualize it, yielding output that shows efficiency and changes in execution status.

[0834] Step 6:

[0835] The terminal performs emotion analysis during user interaction. Inputs include audio data and webcam footage from user conversations. Using technologies from Affectiva and IBM, it analyzes emotions, generates responses based on that data, and provides user-adapted services as output.

[0836] Step 7:

[0837] Users interact with the system using natural language and receive feedback. They input questions and commands in natural language using prompts generated by a generative AI model. The system analyzes these requests, generates responses based on monitoring and sentiment analysis data, and provides them to the user as output. For example, inputting a request such as "Please check the progress of the current task" will provide progress information.

[0838] (Application Example 2)

[0839] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0840] In modern information processing systems, improving operational efficiency and enhancing user experience are crucial challenges. In particular, there is a demand for services that respond to user emotions and individual needs. However, conventional systems lacked sufficient emotion analysis capabilities, making effective communication with users difficult. This resulted in increased workload and inconsistent service quality.

[0841] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0842] In this invention, the server includes means for collecting business data and analyzing that data to visualize the business flow; means for identifying business processes that can be automated from the visualized business flow; means for generating automation blueprints based on the identified business processes; means for importing the generated blueprints into an automation tool and executing the business processes; means for monitoring the executed business processes and generating improvement suggestions; and means for analyzing the user's emotional state and adjusting the response, and for providing individualized responses based on the user's emotional analysis. This enables efficient execution of business processes that take into account the user's emotional state and personalized user responses.

[0843] "Business data" refers to a collection of information and figures related to business processes generated within an organization.

[0844] A "business process flow" is a visual representation of how business processes progress and how data flows.

[0845] An "automation blueprint" is a design document that describes the specific steps and data flow required to automate an identified business process.

[0846] "Automation tools" is a general term for software or systems used to execute automation blueprints.

[0847] "Monitoring" is the act of observing and recording the operation and performance of a business process in progress.

[0848] An "improvement suggestion" is a specific proposal to improve the efficiency and quality of business processes based on monitoring data.

[0849] "Natural language communication" refers to dialogue and information exchange that takes place through the language that humans normally use.

[0850] "Emotional state" refers to the psychological state a user experiences emotionally, and includes changes in emotions that occur under specific circumstances.

[0851] "Individualized support" refers to providing specific support or services tailored to the individual needs and circumstances of each user.

[0852] This invention aims to improve the user experience by incorporating sentiment analysis functionality into a business automation system. The system operates between a server, a terminal, and a user, with each party's role clearly defined.

[0853] First, the server collects business data from the organization's information systems and stores it in a database. The stored business data is analyzed and visualized as a business flow. Data analysis tools (e.g., Tableau or Power BI) can be used as the software for this analysis. Visualization helps identify which business processes are suitable for automation and generates automation blueprints suitable for import into RPA (Robotic Process Automation) tools. These blueprints include process steps, data flow, and error handling.

[0854] The terminal executes business processes based on the generated blueprints. This process utilizes RPA tools installed on the terminal (e.g., UiPath or Automation Anywhere). During process execution, the server monitors its operation and collects performance data. Based on this data, the system generates process improvement suggestions.

[0855] Communication with users is conducted using natural language processing technologies (e.g., NLTK and spaCy). The terminal also uses an emotion analysis engine (e.g., Emotion API) to analyze the user's emotions in real time. If the user is experiencing stress, the system provides considerate responses from the terminal and offers suggestions to reduce their workload.

[0856] As a concrete example, in a system utilizing a home robot, the robot analyzes the user's emotions from their facial expressions and voice, and if it determines that the user is tired, it recommends relaxation music. An example of a prompt message to the generative AI model used in this case would be: "Program music suggestions for when the user is feeling tired on a Friday night. These suggestions should be based on emotion analysis data."

[0857] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0858] Step 1:

[0859] The server collects business data from the organization's information systems. The information obtained as input includes business process log data and error history. The server stores this data in a database, preparing it for subsequent analysis. The output is a well-organized business dataset.

[0860] Step 2:

[0861] The server visualizes the business flow based on the acquired data. It generates a flowchart from the input dataset and visualizes it using data analysis tools. Specifically, it analyzes the start time, end time, and duration of each business step. The output is a visual map showing the flow of the business process.

[0862] Step 3:

[0863] The server identifies business processes that can be automated from the visualized workflow. Machine learning algorithms are used for analysis, picking out repetitive sections within the process. A workflow map is used as input, and the identified automation candidate processes are output.

[0864] Step 4:

[0865] The server generates automation blueprints based on identified business processes. A template-based approach is used for blueprint generation. Detailed information about the identified processes is used as input data, and the output is a blueprint file for use in automation tools.

[0866] Step 5:

[0867] The terminal imports the automation blueprint into the RPA tool and executes the business process. The business is processed automatically according to the specified procedures. The input is the blueprint file, and the output is the automated business result.

[0868] Step 6:

[0869] The server monitors executed business processes and collects performance data. Monitoring tools are used to obtain data such as execution time and error rates. The input is the execution status of the business processes, and the output is a performance report.

[0870] Step 7:

[0871] The device uses an emotion analysis engine to analyze the user's emotional state in real time. It acquires the user's voice and video as input data and performs calculations to identify their emotional state. The output is the analysis result regarding the user's emotions.

[0872] Step 8:

[0873] The device provides personalized responses based on the user's sentiment analysis results. Based on the analysis, it generates appropriate responses and suggestions and presents them to the user. In this process, the sentiment analysis results are used as input, and the output is specific actions or suggestions.

[0874] Step 9:

[0875] Users interact with the system using natural language communication methods provided by their devices. Appropriate responses and improvement suggestions are provided based on user input. Input is the user's natural language, and output is relevant information and suggestions.

[0876] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0877] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0878] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0879] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0880] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0881] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0882] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0883] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0884] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0885] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0886] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0887] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0888] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0889] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0890] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0891] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0892] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0893] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0894] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0895] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0896] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0897] The following is further disclosed regarding the embodiments described above.

[0898] (Claim 1)

[0899] A means of collecting business data, analyzing that data, and visualizing business flows,

[0900] A means of identifying business processes that can be automated from visualized business flows,

[0901] A means for generating automation blueprints based on identified business processes,

[0902] The generated blueprints are imported into an automation tool, and the means of executing the business process are provided.

[0903] A means of monitoring executed business processes and generating improvement suggestions,

[0904] A means of communicating with users using natural language,

[0905] A system that includes this.

[0906] (Claim 2)

[0907] The system according to claim 1, comprising means for evaluating the effectiveness of a business process based on pre-set indicators during the execution of that process.

[0908] (Claim 3)

[0909] The system according to claim 1, comprising means for providing improvement suggestions based on monitoring data in response to user inquiries.

[0910] "Example 1"

[0911] (Claim 1)

[0912] A means of collecting business information from information resources, analyzing that information, and visualizing business procedures,

[0913] A means of identifying business procedures that can be automated from visualized business procedures,

[0914] A means for generating an automation plan based on identified business procedures,

[0915] The generated plan is incorporated into automation technology, and the means of executing business procedures are provided.

[0916] A means of monitoring executed business procedures and generating improvement proposals,

[0917] A means of engaging in natural language dialogue with users,

[0918] A system that includes this.

[0919] (Claim 2)

[0920] The system according to claim 1, comprising means for evaluating the effectiveness of a procedure based on pre-set criteria in the execution of a business procedure.

[0921] (Claim 3)

[0922] The system according to claim 1, comprising means for providing improvement suggestions based on monitoring information in response to user inquiries.

[0923] "Application Example 1"

[0924] (Claim 1)

[0925] A means of collecting business data, analyzing that data, and visualizing business processes,

[0926] A means of identifying automatable work procedures from visualized business processes,

[0927] A means for generating an automation plan based on identified work procedures,

[0928] A means for importing the generated plan into an automated device and executing the work procedure,

[0929] A means of monitoring the executed work procedures and generating improvement suggestions,

[0930] A means of engaging in natural language dialogue with users,

[0931] A means to streamline the management of goods receiving and dispatching within logistics facilities and the arrangement of deliveries,

[0932] A system that includes this.

[0933] (Claim 2)

[0934] The system according to claim 1, comprising means for evaluating the effectiveness of a procedure based on pre-set criteria during the execution of a work procedure.

[0935] (Claim 3)

[0936] The system according to claim 1, comprising means for providing improvement suggestions based on monitoring data in response to user inquiries.

[0937] "Example 2 of combining an emotion engine"

[0938] (Claim 1)

[0939] A means of collecting business data, analyzing that data, and visualizing business processes,

[0940] A means of identifying tasks that can be automated from visualized business processes,

[0941] Means for generating automation designs based on identified tasks,

[0942] The generated design is imported into a process automation tool, and the means of executing business tasks are provided.

[0943] A means of monitoring completed business tasks and generating improvement suggestions,

[0944] A means of analyzing the user's emotional state,

[0945] Means for adjusting service responses based on analyzed sentiment data,

[0946] A means of engaging in natural language dialogue with the user,

[0947] A system that includes this.

[0948] (Claim 2)

[0949] The system according to claim 1, comprising means for evaluating performance based on pre-set criteria in the execution of business tasks.

[0950] (Claim 3)

[0951] The system according to claim 1, comprising means for providing improvement suggestions based on monitoring data and sentiment analysis data in response to user inquiries.

[0952] "Application example 2 when combining with an emotional engine"

[0953] (Claim 1)

[0954] A means of collecting business data, analyzing that data, and visualizing business flows,

[0955] A means of identifying business processes that can be automated from visualized business flows,

[0956] A means for generating automation blueprints based on identified business processes,

[0957] The generated blueprints are imported into an automation tool, and the means of executing the business process are provided.

[0958] A means of monitoring executed business processes and generating improvement suggestions,

[0959] A means of communicating with users using natural language,

[0960] The system includes means for analyzing the user's emotional state and adjusting the response, and means for providing individualized support based on the user's emotional analysis.

[0961] A system that includes this.

[0962] (Claim 2)

[0963] The system according to claim 1, further comprising means for evaluating the effectiveness of a business process based on pre-set indicators during the execution of a business process, and additional means for making personalized suggestions based on the user's emotional state.

[0964] (Claim 3)

[0965] The system according to claim 1, comprising means for providing improvement suggestions based on monitoring data and sentiment analysis results in response to user inquiries. [Explanation of Symbols]

[0966] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of collecting business data, analyzing that data, and visualizing business processes, A means of identifying automatable work procedures from visualized business processes, A means for generating an automation plan based on identified work procedures, A means for importing the generated plan into an automated device and executing the work procedure, A means of monitoring the executed work procedures and generating improvement suggestions, A means of engaging in natural language dialogue with users, A means to streamline the management of goods receiving and dispatching within logistics facilities and the arrangement of deliveries, A system that includes this.

2. The system according to claim 1, comprising means for evaluating the effectiveness of a procedure based on pre-set criteria during the execution of a work procedure.

3. The system according to claim 1, comprising means for providing improvement suggestions based on monitoring data in response to user inquiries.

Images