system

Abstract

Provide a system. 【Solution means】 Recording means for recording user operations, Analysis means for analyzing the information obtained by the recording means, Decomposition means for decomposing the work process based on the result analyzed by the analysis means, Detection means for detecting the room for work efficiency improvement based on the process decomposed by the decomposition means, Generation means for generating a work flow based on the room for efficiency improvement, Design and development means for designing and developing an AI agent based on the generated work flow, Distribution means for distributing the AI agent to the user's device, Communication means for compressing and encrypting the recorded data and transmitting it to the server, Execution means having a control function for automating the work in the factory, A system including the above.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Many modern white-collar jobs contain many repetitive tasks, but the efficiency of individual jobs has not been sufficiently improved. One of the reasons is that expertise and skills in business automation using AI technology are required, and it is not clear which jobs should be selected and optimized. As a result, white-collar workers are burdened every time they automate, and it has become difficult to concentrate on jobs that generate high added value. An object of this invention is to provide a means capable of efficiently automating operations and realizing productivity improvement even when a user does not have expertise in job selection or AI agent development.

Means for Solving the Problems

[0005] This invention relates to a system that includes recording means for recording user operations, analysis means for analyzing data obtained by the recording means, and means for decomposing business processes from the data and detecting areas for improvement. Based on the analysis results, it enables the generation of business flows and the design and development of AI agents. Furthermore, by distributing this AI agent to the user's terminal to realize business automation, users can streamline their work and concentrate on higher value-added tasks without requiring special technical skills.

[0006] A "user" is an individual or organization that uses the system to record business processes and utilizes the automation capabilities of the AI ​​agent.

[0007] "Recording means" refers to a function or device for recording and collecting user operations, and plays a role in acquiring data on business processes.

[0008] "Analysis means" refers to a function or device that analyzes data obtained by recording means, enabling the automatic identification of user operations and the breakdown of business processes.

[0009] A "decomposition method" is a function that divides a business process into individual steps based on the analysis results, enabling a detailed understanding of the process.

[0010] The "detection means" is a function that automatically identifies areas for efficiency improvement from the broken-down processes, and plays a role in extracting points for business improvement.

[0011] The "generation method" is a function that automatically creates an optimized business flow based on the detection results, providing a model for improving the current business process.

[0012] "Design and development methods" refer to the functions or processes for designing and developing AI agents based on the generated business flow, and play a role in realizing the automation of the user's business processes.

[0013] "Distribution means" refers to a function for supplying the developed AI agent to the user's terminal, enabling the developed agent to be adapted to actual work.

[0014] A "system" refers to a technical configuration that comprehensively performs the recording, analysis, optimization, and automation of business processes by combining the above-mentioned means. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention provides a system for automating user tasks by recording PC screen operations and using AI to analyze and streamline business processes. This system consists of a user, a terminal, and a server.

[0037] 1. User actions

[0038] The user launches a dedicated application and records their screen operations during work. This generates data that includes the user's daily work activities.

[0039] 2. Terminal Processing

[0040] The device performs the necessary compression and conversion of the recorded data before sending it to the server. This process can also include encryption to ensure data security.

[0041] 3. Server analysis

[0042] The server analyzes the received data. Here, AI technology is used to identify user actions and break down business processes step by step. The analysis results include the detection of patterns in actions and recurring tasks.

[0043] 4. Generating business workflows

[0044] Based on the analysis results, the server automatically creates the current business process (As-Is flow) and the optimized business process (To-Be flow). This process is presented visually to the user, showing which tasks can be automated.

[0045] 5. Design and Development of AI Agents

[0046] The server designs an AI agent based on the To-Be business flow and automatically generates code. This agent is a program designed to automate the user's business processes and efficiently handle routine tasks.

[0047] 6. Agent Distribution and Implementation

[0048] The generated AI agent is distributed to the device and works in conjunction with the applications the user uses. This agent allows users to reduce the effort required for repetitive tasks, freeing up more time for other creative work.

[0049] Specific example

[0050] For example, for a user who performs data entry tasks daily, this system can record the process of the user manually entering data, and the AI ​​analyzes the entire process. As a result, an automated data entry agent can be designed, enabling daily data entry tasks to be performed with less time and effort.

[0051] As a result, even users without a technical background can easily streamline their work and dramatically improve operational efficiency.

[0052] The following describes the processing flow.

[0053] Step 1:

[0054] The user launches a recording application and records the screen operations of a business process. At this time, the user selects a business process that includes a specific operation and starts recording.

[0055] Step 2:

[0056] The terminal receives the recorded data, compresses it according to the data format, encrypts it as needed, and prepares it for appropriate transfer to the server.

[0057] Step 3:

[0058] The server receives the recorded data sent from the terminal. It decompresses the received data and converts it into a format that can be analyzed by AI.

[0059] Step 4:

[0060] The AI ​​on the server sequentially analyzes the recorded data, classifying and identifying specific user actions such as mouse clicks and keyboard inputs. This allows it to understand the steps that make up the business process.

[0061] Step 5:

[0062] Based on the analysis results, the server breaks down the identified business processes into steps. This breakdown particularly marks repetitive tasks and monotonous work areas.

[0063] Step 6:

[0064] The server automatically generates the current business flow (As-Is) and the optimized business flow (To-Be) based on the broken-down business processes. The generated flows are output in a visual flowchart format that is easy for users to understand.

[0065] Step 7:

[0066] The server designs an AI agent based on the To-Be business flow and generates an automation program to execute it. This program utilizes specific scripts and APIs to automate business processes.

[0067] Step 8:

[0068] The server packages the generated AI agent and performs the distribution procedure to the terminal. This prepares the user's terminal to accept the AI ​​agent.

[0069] Step 9:

[0070] Users run the AI ​​agent on their device and observe how their work processes are automated. If necessary, users can fine-tune the agent's behavior to optimize work efficiency.

[0071] (Example 1)

[0072] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0073] Traditional business automation systems often lack the ability to accurately record and analyze user actions and propose efficient work procedures. Furthermore, most systems require complex configurations, making them difficult for users without a technical background to utilize. There is also a need to effectively identify areas for business efficiency improvements and to rapidly develop and distribute automation programs.

[0074] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0075] In this invention, the server includes an analysis means for analyzing the information and identifying the operation content, a decomposition means for breaking down the work procedure based on the results of the analysis, and a detection means for detecting areas where the operation can be made more efficient based on the procedure. This enables accurate analysis of the user's work procedure, automatic detection of parts that can be made more efficient, and rapid development and distribution of AI automation programs.

[0076] "Recording means" refers to a device or function that electronically captures user behavior and stores that data.

[0077] "Processing means" refers to devices or functions that compress and encrypt the obtained information, enabling secure and efficient data transfer.

[0078] "Transmission means" refers to communication functions or devices used to send processed information to a server.

[0079] "Analysis means" refers to functions or algorithms that analyze information received by the server and identify the user's actions.

[0080] A "disassembly means" refers to a device or function used to break down a work procedure in detail based on the analyzed operation content.

[0081] "Detection means" refers to devices or functions used to identify areas for improvement in business procedures that have been broken down.

[0082] A "generation means" refers to a device or function that creates new work procedures based on the detected areas for improvement.

[0083] "Design and development means" refers to the functions and devices used to design and develop AI automation programs based on generated business procedures.

[0084] "Distribution means" refers to the system or function for providing and installing the developed AI automation program on the user's terminal.

[0085] This invention is a system aimed at automating business processes, providing a series of processes for recording and analyzing user operations and generating automation programs. Specific embodiments are described below.

[0086] Users launch a dedicated application on their work terminal to record their operations during work. This recording meticulously captures the user's mouse movements, keyboard input, and screen interactions. For example, it can capture how routine data entry tasks are performed.

[0087] The terminal processes the recorded operation data and compresses and encrypts it to reduce data size and ensure security. This makes it possible to transmit data to the server efficiently and securely. Specifically, the AES encryption method is often used to enhance security.

[0088] Data is sent to a server, which uses an AI model to analyze user actions based on the received data and breaks down the content into step-by-step business procedures. The server utilizes machine learning algorithms to detect repetitive operation patterns and procedures that can be made more efficient. Based on the analysis results, a newly optimized business procedure (To-Be flow) is automatically generated.

[0089] The server designs and develops AI automation programs (agents) based on the generated business procedures. These agents automate the user's business processes and efficiently handle repetitive tasks. The designed and developed programs are distributed to the user's terminal and work in conjunction with the applications the user uses daily. This frees the user from simple data entry tasks, allowing them to dedicate more time to more creative work.

[0090] A concrete example is the automation of data entry tasks that users perform on a daily basis. Using this system, the AI ​​analyzes the input process and generates a prompt message such as, "Please record all the steps necessary for the user to streamline data entry and design a process to automate it," and an automated input agent is developed. By inputting this example prompt message, it becomes easy to analyze and automate other similar business flows.

[0091] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0092] Step 1:

[0093] Users launch a dedicated recording tool within their business applications to record their daily work operations. Specifically, the software captures the user's input device operations (mouse clicks, keyboard input, etc.) and saves them as data in chronological order. This operation history data is the input, and the output is the recorded operation data file.

[0094] Step 2:

[0095] The terminal compresses and encrypts user-generated operation data. In this process, the operation data file is used as input, and the data size is reduced by a compression algorithm. Furthermore, data security is ensured using encryption technologies such as AES. The output is the compressed and encrypted data file.

[0096] Step 3:

[0097] The terminal sends compressed and encrypted data files to the server. Here, a communication protocol (e.g., HTTPS) is used to ensure secure and reliable data transfer. The input is the compressed and encrypted data files, and the output is the data stored on the server.

[0098] Step 4:

[0099] The server decompresses and decrypts the received data, preparing it for analysis. The decompressed and decrypted data is the input, which restores the original operation data, making it possible to analyze it with the AI ​​model. The output is the operation data converted into an analyzable state.

[0100] Step 5:

[0101] The server uses an AI model to analyze operation data and identify user actions. Machine learning algorithms extract operation features and analyze each step to reveal the work procedure. The input is analyzable operation data, and the output is a set of identified operation patterns and steps.

[0102] Step 6:

[0103] The server breaks down business procedures based on the analysis results and detects areas for improvement. It utilizes process analysis tools to identify which parts can be made more efficient. The input is the identified operation patterns, and the output is the broken-down business procedures and areas for improvement.

[0104] Step 7:

[0105] The server generates new business procedures based on improvement points and designs and develops AI automation programs. It generates code in programming languages ​​such as Python and Java (registered trademark) for the parts that can be automated. The input is the decomposed business procedures and improvement points, and the output is the designed and developed AI agent.

[0106] Step 8:

[0107] The generated AI agent is distributed to the terminal and runs in conjunction with the user's business application. The user confirms that their daily tasks are automated by the implemented AI agent. The input is the AI ​​agent, and the output is the automated business process.

[0108] (Application Example 1)

[0109] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0110] Traditional factory operations often involve a significant amount of manual labor, leaving many areas unautomated. This results in inefficient work processes and a heavy burden on workers, necessitating increased efficiency and automation of work processes. Furthermore, the higher the level of specialization required for a task, the more difficult it becomes to standardize it, posing a significant hurdle to automation.

[0111] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0112] In this invention, the server includes recording means for recording user operations, communication means for compressing and encrypting the recorded data and transmitting it to the server, and execution means having control functions for automating factory operations. This enables efficient analysis and automation of factory work processes, thereby improving work efficiency and reducing the burden on workers.

[0113] "Means of recording user operations" refer to devices or software for saving user work content and procedures in a digital format.

[0114] "A communication method for compressing and encrypting recorded data and sending it to a server" refers to a technology for sending collected data to a remote server in a reduced size and with improved security.

[0115] "Execution means with control functions for automating factory work" refers to technology that executes instructions and operations to mechanically replace manual work in a factory's manufacturing process.

[0116] A "server" is a digital device that receives recorded data, analyzes it, and performs the calculations and instructions necessary to optimize the automated process.

[0117] The system for realizing this invention consists of three main elements: a user, a terminal, and a server. The user uses recording means to record their operations. For example, by utilizing PC screen recording software, the user can collect the procedures for their daily work as digital data.

[0118] The terminal receives the recorded data, compresses and encrypts it for communication, and then reliably transmits it to the server. The common AES encryption method is often used to ensure data security.

[0119] The server operates in a high-performance computing environment and analyzes the transmitted data using generative AI models. Machine learning frameworks such as TENSORFLOW® and PyTorch are used during the analysis process. The server breaks down the user's operational tasks and identifies areas that can be optimized. These specific operational procedures are then implemented in robots to automate factory operations.

[0120] As a concrete example, consider the parts sorting and assembly processes on a factory's production line. By introducing this system, manual work procedures are recorded, analyzed by a server, and parts that can be automated are designed as AI agents. These agents are then incorporated into robots within the factory, significantly improving work efficiency.

[0121] By utilizing the operation of the generated program, users can focus on more creative tasks. Examples of prompts using the generated AI model include, "Analyze the sorting procedure for this part and design the optimal automation flow," and "Generate a program for the robot to maximize the efficiency of a specific assembly line."

[0122] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0123] Step 1:

[0124] The user records the steps taken during the business process. Specifically, they launch PC screen recording software and capture the operations in digital format. The input is the user's operation information, and the output is the recorded data.

[0125] Step 2:

[0126] The terminal compresses and encrypts the recorded data from the user. The input is the recorded data, and the output is the compressed, encrypted data. Here, the data is made into a size that can be efficiently transmitted, and security is enhanced using AES or similar methods.

[0127] Step 3:

[0128] The terminal sends compressed and encrypted data to the server. The input is compressed and encrypted data, and the output is securely transferred data. The data is transferred to the cloud server using a communication protocol.

[0129] Step 4:

[0130] The server decrypts the received data and prepares it for analysis. The input is compressed encrypted data, and the output is the decrypted data for analysis.

[0131] Step 5:

[0132] The server uses a generated AI model to analyze the user's work process. The input is decoded analysis data, and the output is information on the decomposition and optimization of the business process. Analysis is performed using tools such as TensorFlow and PyTorch.

[0133] Step 6:

[0134] The server designs an optimized business workflow based on the analysis results. The input is decomposition information of the business process, and the output is the To-Be flow. The server visually presents procedures that can be made more efficient.

[0135] Step 7:

[0136] The server automatically generates the design and code for the AI ​​agent based on the To-Be flow. The input is the To-Be flow, and the output is the implementation code for the AI ​​agent. The generated code is specialized for robotics.

[0137] Step 8:

[0138] The server distributes AI agents to the terminals. The input is the implementation code of the AI ​​agent, and the output is the distribution status to the user's device.

[0139] Step 9:

[0140] Users operate robots using the received AI agent to automate factory operations. The input is the AI ​​agent, and the output is the automated factory work. Based on the agent, the robot automatically performs the streamlined tasks.

[0141] In each of the above steps, measures have been taken to ensure that the creation and analysis of prompt sentences using the generative AI model proceeds smoothly.

[0142] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0143] This invention provides a system that combines a system for streamlining user business processes by recording and analyzing PC screen operations with an emotion engine that recognizes user emotions. This system consists of a user, a terminal, and a server, and provides advanced support for automating user tasks.

[0144] 1. User actions

[0145] The user launches a dedicated application to begin their work. The workflow is recorded, and the user's actions are captured in real time. This recording includes emotional information such as the user's voice, facial expressions, and heart rate.

[0146] 2. Terminal Processing

[0147] The device compresses recorded operational and emotional data in real time and securely transmits it to the server. Emotional data can be anonymized to protect individual privacy.

[0148] 3. Server analysis

[0149] The server analyzes the transmitted operation data. This analysis uses AI technology to break down the user's actions into smaller parts. In addition, an emotion engine analyzes the user's emotional state, making it possible to quantify it as stress, satisfaction, and other factors.

[0150] 4. Optimization based on business flow and sentiment analysis

[0151] The server analyzes operational data and incorporates emotional data to consider how to optimize business processes. This allows for optimization that can reduce user psychological stress and improve satisfaction. The generated business flow is adjusted to reflect the user's emotional state.

[0152] 5. Design and Development of AI Agents

[0153] The server designs AI agents based on optimized workflows. These agents can not only automate actual tasks but also adapt their responses to the user's emotions.

[0154] 6. Agent Distribution and Implementation

[0155] The designed AI agent is distributed to the device and operates in the user's environment. The agent is integrated into the user's workflow and provides emotion-based, optimal support.

[0156] Specific example

[0157] For example, in the case of a support staff member who frequently interacts with customers, this system can automatically suggest optimization measures such as reducing workload or sending break notifications when the emotion engine detects that stress levels are high. This improves the ease of work for the staff member and can lead to increased long-term productivity.

[0158] Through the above, this system adds a new element—emotion—to business automation, enabling the proposal of work styles that are more considerate of human psychological states.

[0159] The following describes the processing flow.

[0160] Step 1:

[0161] The user launches a dedicated application and begins their work. The application records the user's screen activity and simultaneously records emotional data using a microphone, camera, and heart rate sensor.

[0162] Step 2:

[0163] The device compresses recorded operation data and emotion data in real time and encrypts the data to ensure security. It then prepares the data for transmission to the server.

[0164] Step 3:

[0165] The server analyzes the received operation data and uses AI to break down the user's work actions. Specifically, it identifies actions such as mouse clicks, keyboard input, and window switching.

[0166] Step 4:

[0167] The server utilizes an emotion engine to analyze the user's emotional state from their facial expressions, voice, heart rate, and other data. Based on these results, it quantifies the user's stress level and fatigue level.

[0168] Step 5:

[0169] The server integrates analysis results of operational data with emotional data to optimize business processes and user psychological states. This includes generating improvement suggestions to adjust work processes when the workload is high or when emotional data indicates high stress levels.

[0170] Step 6:

[0171] The server automatically generates a workflow as the "To-Be" state, taking into account emotions and efficiency. It creates a workflow that considers stress reduction so that users can perform their tasks with peace of mind.

[0172] Step 7:

[0173] The server designs an AI agent based on the newly generated To-Be workflow and creates code to provide the optimal work environment for the user's psychological state. This agent includes scripts that adjust tasks according to the user's state.

[0174] Step 8:

[0175] Agents are distributed to terminals, and users run these agents. Based on sentiment analysis, the agents automatically review task assignments, change task priorities as needed, and notify users.

[0176] Step 9:

[0177] Users can verify that their tasks are automatically adjusted by the AI ​​agent and provide feedback on their actions approximately three times a day (morning, noon, and evening) based on their emotional changes. This feedback allows the agent to learn further and use it to make adjustments in the future.

[0178] (Example 2)

[0179] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0180] Traditional business automation systems focused on efficiency improvements to enhance user productivity, but they lacked sufficient optimization that considered user emotional states. As a result, reducing user psychological stress and improving job satisfaction were not adequately achieved. Therefore, there is a need for systems that not only improve operational efficiency but also optimize by reflecting user emotional states in real time.

[0181] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0182] In this invention, the server includes acquisition means for recording user operations and emotional states, transmission means for compressing and transmitting the data obtained by the acquisition means, and analysis means for analyzing the data transmitted by the transmission means and decomposing work steps and emotional states. This makes it possible to streamline the user's workflow while reducing psychological stress and improving job satisfaction.

[0183] "User actions and emotional state" refers to the state of the user, including both physical actions (keyboard input, mouse operation, etc.) and psychological responses such as heart rate and facial expressions.

[0184] "Acquisition means" refers to mechanical or software devices for collecting and recording user actions and emotional states.

[0185] "Transmission means" refers to a communication function that compresses acquired data into a predetermined format and securely transfers it to a server.

[0186] "Analysis means" refers to an algorithm or device that uses transmitted data to subdivide business processes and evaluate the user's emotional state.

[0187] "Design and generation methods" refer to processes or functions that streamline business workflows and automatically generate AI agents based on data obtained through analysis.

[0188] "Distribution means" refers to the function of providing the designed and generated AI agent to the user's device and ensuring its proper installation and execution.

[0189] A "business process flow" refers to a planned set of steps or a set of procedures designed to optimize a user's business processes.

[0190] This invention is a system for optimizing work processes by recording user actions and emotional states. Users begin their work using a dedicated application. The application, installed on a desktop or laptop computer, captures user actions and emotional states in real time. Specifically, it collects user interactions such as keyboard input and mouse operations, as well as emotional information such as heart rate and facial expressions. This data is acquired through wearable devices and cameras.

[0191] The terminal compresses the collected data and sends it to the server using a secure communication protocol. Commonly used compression algorithms (e.g., H.264 encoding or ZIP compression) are employed for data compression, and SSL / TLS encryption is used for communication.

[0192] The server analyzes the transmitted data using advanced AI algorithms. This analysis breaks down business processes and identifies common work patterns. It also quantifies users' psychological states from emotional data and evaluates stress and satisfaction levels. A cloud-based AI platform is used for the analysis.

[0193] The data analyzed on the server is used to optimize business workflows. This optimization includes adjustments to reduce user stress and suggestions for more efficient work sequences. This enables the creation of a more comfortable and productive work environment.

[0194] As a concrete example, when a support staff member who frequently interacts with customers uses this system, if the emotion engine determines that the user is experiencing high stress levels, it will automatically suggest taking a break and provide other support. The AI ​​agent generated for this purpose is designed and developed in Python or Java and distributed to the user's device.

[0195] An example of a prompt to input into a generative AI model might be, "Explain how to use an emotion engine to detect stress levels in a sales team and design automated responses." Such prompts allow the system to automatically analyze the user's work and derive solutions.

[0196] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0197] Step 1:

[0198] The user launches a dedicated application to begin work. The application acquires the user's keyboard input and mouse operations, as well as heart rate and facial recognition camera data received from connected wearable devices, in real time. Input data includes keyboard and mouse event logs, heart rate, and facial expression data. This data is output as a log organized by time and operation using the application's built-in functions.

[0199] Step 2:

[0200] The terminal compresses the log data acquired in Step 1 and prepares it for efficient transfer to the server. Specifically, it encodes image data in JPEG format, video data in H.264 format, and compiles event logs in JSON format. These compressed datasets are encrypted using SSL / TLS and securely sent to the server. The input is the log data obtained in Step 1, and the output is the compressed and encrypted data for transmission.

[0201] Step 3:

[0202] The server receives data transmitted from the terminal and performs analysis using an AI algorithm. During data analysis, it thoroughly analyzes the operation details and models the workflow. Furthermore, an emotion engine quantifies the user's emotional state from heart rate and facial expression data. At this stage, the input is compressed and encrypted transmitted data, which is then decompressed and analyzed to output the breakdown of business steps and numerical values ​​representing the emotional state.

[0203] Step 4:

[0204] The server optimizes the business workflow based on the analysis results. It identifies steps where efficiency can be improved according to predefined criteria and constructs a new business process that takes into account the user's emotional state. In this process, the input consists of the business step data and quantified emotional information obtained in step 3, while the output is a model of the optimized business workflow.

[0205] Step 5:

[0206] Based on the optimized workflow, the server automatically designs and generates an AI agent. The AI ​​agent is coded using a programming language and tested and verified on the server side. The input is the workflow obtained in step 4, and the output is the code for a workable AI agent.

[0207] Step 6:

[0208] The server distributes the designed and generated AI agent to the terminal and deploys it into the user's work environment. The agent is integrated into the system and provides the user with timely, emotion-based support. The input is the AI ​​agent code obtained in step 5, and the output is the operational agent implemented on the user's terminal.

[0209] (Application Example 2)

[0210] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0211] In the field of elderly care, staff workloads and stress levels are often increasing, leading to challenges such as decreased work efficiency and a decline in the quality of care. To address this, there is a need for systems that streamline work processes while considering the emotional state of staff.

[0212] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0213] In this invention, the server includes recording means for recording user operations, analysis means for analyzing data, and emotion analysis means for recognizing user emotions. This makes it possible to improve work efficiency while reducing stress for care workers.

[0214] A "user" is a person who operates a system and is an executor in a business process—a person who uses an information processing device.

[0215] "Recording means" refers to a device or software that has the function of acquiring user operation information and storing it as data in a format that can be analyzed later.

[0216] "Analysis means" refers to a device or software that analyzes data obtained by recording means and extracts information for subdividing business processes.

[0217] "Decomposition means" refers to methods or apparatus for decomposing a business process into its elements based on the results of analysis performed by analytical means.

[0218] "Detection means" refers to methods or devices for identifying parts of a process that can be made more efficient based on the process that has been disassembled by the disassembly means.

[0219] "Generation means" refers to methods or devices for creating new business workflows based on the areas for improvement identified by the detection means.

[0220] "Design and development means" refers to methods and devices for constructing a support system to provide work assistance based on the generated business flow.

[0221] "Distribution means" refers to methods or devices for providing support systems developed through design and development means to each user's information processing device.

[0222] "Emotional analysis means" refers to a device or software that has the function of identifying a user's emotional state and converting that information into a format that can be used to improve work efficiency.

[0223] "Emotion-based metrics" are data obtained by quantifying or evaluating user emotional information acquired through emotion analysis methods, and are used to improve work efficiency.

[0224] This invention is a system aimed at streamlining the work of staff in caregiving settings while simultaneously reducing their mental burden. This system uses smart glasses, which are information processing devices worn by the user. The device is equipped with recording means for recording the user's operations and actions. The recording means has the function of acquiring the staff member's voice, movements, and related biometric data.

[0225] The device transmits recorded data to the server via a secure protocol. The data is compressed and encrypted in real time, ensuring security.

[0226] The server leverages AWS® cloud services to efficiently analyze large amounts of data. It uses multiple data analysis algorithms written in Python. The server breaks down the transmitted data into multiple elements and also analyzes the user's emotional state. Google®'s Sentiment Analysis API is used for this sentiment analysis.

[0227] Based on the analysis results, the server designs and develops a work support system. The interactive work support system is designed to assist the user based on the generated workflow and emotional state. Depending on the design and development method, it may be possible to prompt users experiencing high stress levels to take short breaks.

[0228] Ultimately, the support system is distributed to the device and assists the user's work processes. For example, if emotion analysis indicates that a staff member working in a nursing home is experiencing increased stress during routine tasks, they will receive a notification prompting them to take a break at an appropriate time. This is expected to improve the quality of care and reduce the workload on staff.

[0229] An example of a prompt to input into the generating AI model would be: "We are developing a system to streamline users' business processes. The system will recognize users' emotions and use that information to optimize operations. We are seeking specific application proposals that take into account applications in nursing care settings."

[0230] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0231] Step 1:

[0232] When a user begins work, the device's recording capabilities acquire user actions, voice, movements, and biometric data such as heart rate in real time. Input consists of user actions and biometric information, while output is compressed data of this information. Data collection is performed using sensors within the smart glasses.

[0233] Step 2:

[0234] The terminal compresses the recorded data in real time and sends it to the server using a secure protocol. The input is the raw data before compression, and the output is encrypted data sent to the server. Here, preparations are made to securely send the data to the server using encryption algorithms such as AES.

[0235] Step 3:

[0236] The server receives the transmitted data and analyzes it using analytical tools. The input is encrypted data, and the output is the analyzed business process data. First, the data is decrypted, and an analysis algorithm using Python is executed to identify the business flow and emotional state.

[0237] Step 4:

[0238] Based on the analysis results, the server uses a decomposition tool to break down the business process into specific elements. The input is the analysis results, and the output is a list of the decomposed business processes. The business process is divided into individual tasks, and the possibility of improvement is evaluated for each task.

[0239] Step 5:

[0240] The server uses a generation method to link decomposed information with sentiment data and design an optimized workflow. The input is metric data obtained from sentiment analysis and information on decomposed business processes, and the output is the optimized new workflow. It processes sentiment data using Google's sentiment analysis API and automatically suggests stress reduction and efficient work assignments.

[0241] Step 6:

[0242] The server develops a work support system aligned with the user's workflow through design and development tools, and distributes it to the terminal. The input is the optimized workflow, and the output is the customized program incorporated into the distributed work support system. Software updates are performed on the terminal, and notifications are configured to be displayed on the smart glasses' display.

[0243] Step 7:

[0244] The user's terminal accepts the distributed work support system and provides support information in real time during work based on that system. Input is data from the distributed support system, and output is notifications and alerts for work improvement displayed to the user. For example, if the stress level is high, a message recommending a 10-minute break will be displayed on the screen.

[0245] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0246] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0247] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0248] [Second Embodiment]

[0249] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0250] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0251] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0252] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0253] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0254] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0255] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0256] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0257] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0258] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0259] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0260] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0261] This invention provides a system for automating user tasks by recording PC screen operations and using AI to analyze and streamline business processes. This system consists of a user, a terminal, and a server.

[0262] 1. User actions

[0263] The user launches a dedicated application and records their screen operations during work. This generates data that includes the user's daily work activities.

[0264] 2. Terminal Processing

[0265] The terminal performs the necessary compression and conversion of the recorded data before sending it to the server. This process can also include encryption to ensure data security.

[0266] 3. Server analysis

[0267] The server analyzes the received data. Here, AI technology is used to identify user actions and break down business processes step by step. The analysis results include the detection of patterns in actions and recurring tasks.

[0268] 4. Generating business workflows

[0269] Based on the analysis results, the server automatically creates the current business process (As-Is flow) and the optimized business process (To-Be flow). This process is presented visually to the user, showing which tasks can be automated.

[0270] 5. Design and Development of AI Agents

[0271] The server designs an AI agent based on the To-Be business flow and automatically generates code. This agent is a program designed to automate the user's business processes and efficiently handle routine tasks.

[0272] 6. Agent Distribution and Implementation

[0273] The generated AI agent is distributed to the device and works in conjunction with the applications the user uses. This agent allows users to reduce the effort required for repetitive tasks, freeing up more time for other creative work.

[0274] Specific example

[0275] For example, in the case of a user who performs data entry work every day, by using this system, the process of the user manually entering data is recorded, and the entire process is analyzed by AI. As a result, an automatic input agent is designed, enabling the daily data entry work to be executed without spending time and effort.

[0276] As described above, even users without technical backgrounds can easily streamline their work, dramatically improving work efficiency.

[0277] The following describes the processing flow.

[0278] Step 1:

[0279] The user launches the recording application and records the screen operations of the business process. At this time, the user selects the business that includes specific operations and starts recording.

[0280] Step 2:

[0281] The terminal receives the recorded data, compresses it according to the data format and encrypts it if necessary, and prepares to transfer it to the server appropriately.

[0282] Step 3:

[0283] The server receives the recorded data transmitted from the terminal. The received data is decompressed and converted into a format that can be analyzed by AI.

[0284] Step 4:

[0285] The AI on the server sequentially analyzes the recorded data and classifies and identifies specific operations such as the user's mouse clicks and keyboard inputs. By doing this, it grasps the steps that make up the business process.

[0286] Step 5:

[0287] Based on the analysis results, the server breaks down the identified business processes into steps. This breakdown particularly marks repetitive tasks and monotonous work areas.

[0288] Step 6:

[0289] The server automatically generates the current business flow (As-Is) and the optimized business flow (To-Be) based on the broken-down business processes. The generated flows are output in a visual flowchart format that is easy for users to understand.

[0290] Step 7:

[0291] The server designs an AI agent based on the To-Be business flow and generates an automation program to execute it. This program utilizes specific scripts and APIs to automate business processes.

[0292] Step 8:

[0293] The server packages the generated AI agent and performs the distribution procedure to the terminal. This prepares the user's terminal to accept the AI ​​agent.

[0294] Step 9:

[0295] Users run the AI ​​agent on their device and observe how their work processes are automated. If necessary, users can fine-tune the agent's behavior to optimize work efficiency.

[0296] (Example 1)

[0297] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0298] Traditional business automation systems often lack the ability to accurately record and analyze user actions and propose efficient work procedures. Furthermore, most systems require complex configurations, making them difficult for users without a technical background to utilize. There is also a need to effectively identify areas for business efficiency improvements and to rapidly develop and distribute automation programs.

[0299] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0300] In this invention, the server includes an analysis means for analyzing the information and identifying the operation content, a decomposition means for breaking down the work procedure based on the results of the analysis, and a detection means for detecting areas where the operation can be made more efficient based on the procedure. This enables accurate analysis of the user's work procedure, automatic detection of parts that can be made more efficient, and rapid development and distribution of AI automation programs.

[0301] "Recording means" refers to a device or function that electronically captures user behavior and stores that data.

[0302] "Processing means" refers to devices or functions that compress and encrypt the obtained information, enabling secure and efficient data transfer.

[0303] "Transmission means" refers to communication functions or devices used to send processed information to a server.

[0304] "Analysis means" refers to functions or algorithms that analyze information received by the server and identify the user's actions.

[0305] A "disassembly means" refers to a device or function used to break down a work procedure in detail based on the analyzed operation content.

[0306] "Detection means" refers to devices or functions used to identify areas for improvement in business procedures that have been broken down.

[0307] The "generation means" is a device or function that creates new business processes based on the detected room for efficiency improvement.

[0308] The "design and development means" is a function or device for designing and developing an AI automation program based on the generated business process.

[0309] The "distribution means" is a system or function for providing and installing the developed AI automation program on the user's terminal.

[0310] This invention is a system for business automation, which provides a series of processes for recording, analyzing the user's operations, and generating an automation program. The following shows its specific embodiments.

[0311] The user launches a dedicated application on their work terminal to record the operations during work. At this time, the user's mouse operations, keyboard inputs, and screen interactions are recorded in detail. For example, it is possible to capture how daily data entry work is performed.

[0312] The terminal processes the recorded operation data and performs compression and encryption to reduce the data size and ensure security. This enables the data to be transmitted to the server efficiently and securely. Specifically, the AES encryption method is often used to enhance security.

[0313] The data is transmitted to the server. Based on the received data, the server analyzes the user's operations using an AI model and decomposes the content into business processes step by step. The server utilizes machine learning algorithms to detect repetitive operation patterns and procedures that can be optimized. Based on the analysis results, a newly optimized business process (To-Be flow) is automatically generated.

[0314] The server designs and develops AI automation programs (agents) based on the generated business procedures. These agents automate the user's business processes and efficiently handle repetitive tasks. The designed and developed programs are distributed to the user's terminal and work in conjunction with the applications the user uses daily. This frees the user from simple data entry tasks, allowing them to dedicate more time to more creative work.

[0315] A concrete example is the automation of data entry tasks that users perform on a daily basis. Using this system, the AI ​​analyzes the input process and generates a prompt message such as, "Please record all the steps necessary for the user to streamline data entry and design a process to automate it," and an automated input agent is developed. By inputting this example prompt message, it becomes easy to analyze and automate other similar business flows.

[0316] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0317] Step 1:

[0318] Users launch a dedicated recording tool within their business applications to record their daily work operations. Specifically, the software captures the user's input device operations (mouse clicks, keyboard input, etc.) and saves them as data in chronological order. This operation history data is the input, and the output is the recorded operation data file.

[0319] Step 2:

[0320] The terminal compresses and encrypts user-generated operation data. In this process, the operation data file is used as input, and the data size is reduced by a compression algorithm. Furthermore, data security is ensured using encryption technologies such as AES. The output is the compressed and encrypted data file.

[0321] Step 3:

[0322] The terminal sends compressed and encrypted data files to the server. Here, a communication protocol (e.g., HTTPS) is used to ensure secure and reliable data transfer. The input is the compressed and encrypted data files, and the output is the data stored on the server.

[0323] Step 4:

[0324] The server decompresses and decrypts the received data, preparing it for analysis. The decompressed and decrypted data is the input, which restores the original operation data, making it possible to analyze it with the AI ​​model. The output is the operation data converted into an analyzable state.

[0325] Step 5:

[0326] The server uses an AI model to analyze operation data and identify user actions. Machine learning algorithms extract operation features and analyze each step to reveal the work procedure. The input is analyzable operation data, and the output is a set of identified operation patterns and steps.

[0327] Step 6:

[0328] The server breaks down business procedures based on the analysis results and detects areas for improvement. It utilizes process analysis tools to identify which parts can be made more efficient. The input is the identified operation patterns, and the output is the broken-down business procedures and areas for improvement.

[0329] Step 7:

[0330] The server generates new business procedures based on improvement points and designs and develops AI automation programs. It generates code in programming languages ​​such as Python and Java for the parts that can be automated. The input is the decomposed business procedures and improvement points, and the output is the designed and developed AI agent.

[0331] Step 8:

[0332] The generated AI agent is distributed to the terminal and runs in conjunction with the user's business application. The user confirms that their daily tasks are automated by the implemented AI agent. The input is the AI ​​agent, and the output is the automated business process.

[0333] (Application Example 1)

[0334] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0335] Traditional factory operations often involve a significant amount of manual labor, leaving many areas unautomated. This results in inefficient work processes and a heavy burden on workers, necessitating increased efficiency and automation of work processes. Furthermore, the higher the level of specialization required for a task, the more difficult it becomes to standardize it, posing a significant hurdle to automation.

[0336] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0337] In this invention, the server includes recording means for recording user operations, communication means for compressing and encrypting the recorded data and transmitting it to the server, and execution means having control functions for automating factory operations. This enables efficient analysis and automation of factory work processes, thereby improving work efficiency and reducing the burden on workers.

[0338] "Means of recording user operations" refer to devices or software for saving user work content and procedures in a digital format.

[0339] "A communication method for compressing and encrypting recorded data and sending it to a server" refers to a technology for sending collected data to a remote server in a reduced size and with improved security.

[0340] "Execution means with control functions for automating factory work" refers to technology that executes instructions and operations to mechanically replace manual work in a factory's manufacturing process.

[0341] A "server" is a digital device that receives recorded data, analyzes it, and performs the calculations and instructions necessary to optimize the automated process.

[0342] The system for realizing this invention consists of three main elements: a user, a terminal, and a server. The user uses recording means to record their operations. For example, by utilizing PC screen recording software, the user can collect the procedures for their daily work as digital data.

[0343] The terminal receives the recorded data, compresses and encrypts it for communication, and then reliably transmits it to the server. The common AES encryption method is often used to ensure data security.

[0344] The server operates in a high-performance computing environment and analyzes the transmitted data using generative AI models. Machine learning frameworks such as TensorFlow and PyTorch are used during the analysis process. The server breaks down the user's operational tasks and identifies areas that can be optimized. These specific operational procedures are then implemented in robots to automate factory operations.

[0345] As a concrete example, consider the parts sorting and assembly processes on a factory's production line. By introducing this system, manual work procedures are recorded, analyzed by a server, and parts that can be automated are designed as AI agents. These agents are then incorporated into robots within the factory, significantly improving work efficiency.

[0346] By utilizing the operation of the generated program, users can focus on more creative tasks. Examples of prompts using the generated AI model include, "Analyze the sorting procedure for this part and design the optimal automation flow," and "Generate a program for the robot to maximize the efficiency of a specific assembly line."

[0347] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0348] Step 1:

[0349] The user records the steps taken during the business process. Specifically, they launch PC screen recording software and capture the operations in digital format. The input is the user's operation information, and the output is the recorded data.

[0350] Step 2:

[0351] The terminal compresses and encrypts the recorded data from the user. The input is the recorded data, and the output is the compressed, encrypted data. Here, the data is made into a size that can be efficiently transmitted, and security is enhanced using AES or similar methods.

[0352] Step 3:

[0353] The terminal sends compressed and encrypted data to the server. The input is compressed and encrypted data, and the output is securely transferred data. The data is transferred to the cloud server using a communication protocol.

[0354] Step 4:

[0355] The server decrypts the received data and prepares it for analysis. The input is compressed encrypted data, and the output is the decrypted data for analysis.

[0356] Step 5:

[0357] The server uses a generated AI model to analyze the user's work process. The input is decoded analysis data, and the output is information on the decomposition and optimization of the business process. Analysis is performed using tools such as TensorFlow and PyTorch.

[0358] Step 6:

[0359] The server designs an optimized business workflow based on the analysis results. The input is decomposition information of the business process, and the output is the To-Be flow. The server visually presents procedures that can be made more efficient.

[0360] Step 7:

[0361] The server automatically generates the design and code for the AI ​​agent based on the To-Be flow. The input is the To-Be flow, and the output is the implementation code for the AI ​​agent. The generated code is specialized for robotics.

[0362] Step 8:

[0363] The server distributes AI agents to the terminals. The input is the implementation code of the AI ​​agent, and the output is the distribution status to the user's device.

[0364] Step 9:

[0365] Users operate robots using the received AI agent to automate factory operations. The input is the AI ​​agent, and the output is the automated factory work. Based on the agent, the robot automatically performs the streamlined tasks.

[0366] In each of the above steps, measures have been taken to ensure that the creation and analysis of prompt sentences using the generative AI model proceeds smoothly.

[0367] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0368] This invention provides a system that combines a system for streamlining user business processes by recording and analyzing PC screen operations with an emotion engine that recognizes user emotions. This system consists of a user, a terminal, and a server, and provides advanced support for automating user tasks.

[0369] 1. User actions

[0370] The user launches a dedicated application to begin their work. The workflow is recorded, and the user's actions are captured in real time. This recording includes emotional information such as the user's voice, facial expressions, and heart rate.

[0371] 2. Terminal Processing

[0372] The device compresses recorded operational and emotional data in real time and securely transmits it to the server. Emotional data can be anonymized to protect individual privacy.

[0373] 3. Server analysis

[0374] The server analyzes the transmitted operation data. This analysis uses AI technology to break down the user's actions into smaller parts. In addition, an emotion engine analyzes the user's emotional state, making it possible to quantify it as stress, satisfaction, and other factors.

[0375] 4. Optimization based on business flow and sentiment analysis

[0376] The server analyzes operational data and incorporates emotional data to consider how to optimize business processes. This allows for optimization that can reduce user psychological stress and improve satisfaction. The generated business flow is adjusted to reflect the user's emotional state.

[0377] 5. Design and Development of AI Agents

[0378] The server designs AI agents based on optimized workflows. These agents can not only automate actual tasks but also adapt their responses to the user's emotions.

[0379] 6. Agent Distribution and Implementation

[0380] The designed AI agent is distributed to the device and operates in the user's environment. The agent is integrated into the user's workflow and provides emotion-based, optimal support.

[0381] Specific example

[0382] For example, in the case of a support staff member who frequently interacts with customers, this system can automatically suggest optimization measures such as reducing workload or sending break notifications when the emotion engine detects that stress levels are high. This improves the ease of work for the staff member and can lead to increased long-term productivity.

[0383] Through the above, this system adds a new element—emotion—to business automation, enabling the proposal of work styles that are more considerate of human psychological states.

[0384] The following describes the processing flow.

[0385] Step 1:

[0386] The user launches a dedicated application and begins their work. The application records the user's screen activity and simultaneously records emotional data using a microphone, camera, and heart rate sensor.

[0387] Step 2:

[0388] The device compresses recorded operation data and emotion data in real time and encrypts the data to ensure security. It then prepares the data for transmission to the server.

[0389] Step 3:

[0390] The server analyzes the received operation data and uses AI to break down the user's work actions. Specifically, it identifies actions such as mouse clicks, keyboard input, and window switching.

[0391] Step 4:

[0392] The server utilizes an emotion engine to analyze the user's emotional state from their facial expressions, voice, heart rate, and other data. Based on these results, it quantifies the user's stress level and fatigue level.

[0393] Step 5:

[0394] The server integrates analysis results of operational data with emotional data to optimize business processes and user psychological states. This includes generating improvement suggestions to adjust work processes when the workload is high or when emotional data indicates high stress levels.

[0395] Step 6:

[0396] The server automatically generates a workflow as the "To-Be" state, taking into account emotions and efficiency. It creates a workflow that considers stress reduction so that users can perform their tasks with peace of mind.

[0397] Step 7:

[0398] The server designs an AI agent based on the newly generated To-Be workflow and creates code to provide the optimal work environment for the user's psychological state. This agent includes scripts that adjust tasks according to the user's state.

[0399] Step 8:

[0400] Agents are distributed to terminals, and users run these agents. Based on sentiment analysis, the agents automatically review task assignments, change task priorities as needed, and notify users.

[0401] Step 9:

[0402] Users can verify that their tasks are automatically adjusted by the AI ​​agent and provide feedback on their actions approximately three times a day (morning, noon, and evening) based on their emotional changes. This feedback allows the agent to learn further and use it to make adjustments in the future.

[0403] (Example 2)

[0404] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0405] Traditional business automation systems focused on efficiency improvements to enhance user productivity, but they lacked sufficient optimization that considered user emotional states. As a result, reducing user psychological stress and improving job satisfaction were not adequately achieved. Therefore, there is a need for systems that not only improve operational efficiency but also optimize by reflecting user emotional states in real time.

[0406] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0407] In this invention, the server includes acquisition means for recording user operations and emotional states, transmission means for compressing and transmitting the data obtained by the acquisition means, and analysis means for analyzing the data transmitted by the transmission means and decomposing work steps and emotional states. This makes it possible to streamline the user's workflow while reducing psychological stress and improving job satisfaction.

[0408] "User actions and emotional state" refers to the state of the user, including both physical actions (keyboard input, mouse operation, etc.) and psychological responses such as heart rate and facial expressions.

[0409] "Acquisition means" refers to mechanical or software devices for collecting and recording user actions and emotional states.

[0410] "Transmission means" refers to a communication function that compresses acquired data into a predetermined format and securely transfers it to a server.

[0411] "Analysis means" refers to an algorithm or device that uses transmitted data to subdivide business processes and evaluate the user's emotional state.

[0412] "Design and generation methods" refer to processes or functions that streamline business workflows and automatically generate AI agents based on data obtained through analysis.

[0413] "Distribution means" refers to the function of providing the designed and generated AI agent to the user's device and ensuring its proper installation and execution.

[0414] A "business process flow" refers to a planned set of steps or a set of procedures designed to optimize a user's business processes.

[0415] This invention is a system for optimizing work processes by recording user actions and emotional states. Users begin their work using a dedicated application. The application, installed on a desktop or laptop computer, captures user actions and emotional states in real time. Specifically, it collects user interactions such as keyboard input and mouse operations, as well as emotional information such as heart rate and facial expressions. This data is acquired through wearable devices and cameras.

[0416] The terminal compresses the collected data and sends it to the server using a secure communication protocol. Commonly used compression algorithms (e.g., H.264 encoding or ZIP compression) are employed for data compression, and SSL / TLS encryption is used for communication.

[0417] The server analyzes the transmitted data using advanced AI algorithms. This analysis breaks down business processes and identifies common work patterns. It also quantifies users' psychological states from emotional data and evaluates stress and satisfaction levels. A cloud-based AI platform is used for the analysis.

[0418] The data analyzed on the server is used to optimize business workflows. This optimization includes adjustments to reduce user stress and suggestions for more efficient work sequences. This enables the creation of a more comfortable and productive work environment.

[0419] As a concrete example, when a support staff member who frequently interacts with customers uses this system, if the emotion engine determines that the user is experiencing high stress levels, it will automatically suggest taking a break and provide other support. The AI ​​agent generated for this purpose is designed and developed in Python or Java and distributed to the user's device.

[0420] An example of a prompt to input into a generative AI model might be, "Explain how to use an emotion engine to detect stress levels in a sales team and design automated responses." Such prompts allow the system to automatically analyze the user's work and derive solutions.

[0421] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0422] Step 1:

[0423] The user launches a dedicated application to begin work. The application acquires the user's keyboard input and mouse operations, as well as heart rate and facial recognition camera data received from connected wearable devices, in real time. Input data includes keyboard and mouse event logs, heart rate, and facial expression data. This data is output as a log organized by time and operation using the application's built-in functions.

[0424] Step 2:

[0425] The terminal compresses the log data acquired in Step 1 and prepares it for efficient transfer to the server. Specifically, it encodes image data in JPEG format, video data in H.264 format, and compiles event logs in JSON format. These compressed datasets are encrypted using SSL / TLS and securely sent to the server. The input is the log data obtained in Step 1, and the output is the compressed and encrypted data for transmission.

[0426] Step 3:

[0427] The server receives data transmitted from the terminal and performs analysis using an AI algorithm. During data analysis, it thoroughly analyzes the operation details and models the workflow. Furthermore, an emotion engine quantifies the user's emotional state from heart rate and facial expression data. At this stage, the input is compressed and encrypted transmitted data, which is then decompressed and analyzed to output the breakdown of business steps and numerical values ​​representing the emotional state.

[0428] Step 4:

[0429] The server optimizes the business workflow based on the analysis results. It identifies steps where efficiency can be improved according to predefined criteria and constructs a new business process that takes into account the user's emotional state. In this process, the input consists of the business step data and quantified emotional information obtained in step 3, while the output is a model of the optimized business workflow.

[0430] Step 5:

[0431] Based on the optimized workflow, the server automatically designs and generates an AI agent. The AI ​​agent is coded using a programming language and tested and verified on the server side. The input is the workflow obtained in step 4, and the output is the code for a workable AI agent.

[0432] Step 6:

[0433] The server distributes the designed and generated AI agent to the terminal and deploys it into the user's work environment. The agent is integrated into the system and provides the user with timely, emotion-based support. The input is the AI ​​agent code obtained in step 5, and the output is the operational agent implemented on the user's terminal.

[0434] (Application Example 2)

[0435] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0436] In the field of elderly care, staff workloads and stress levels are often increasing, leading to challenges such as decreased work efficiency and a decline in the quality of care. To address this, there is a need for systems that streamline work processes while considering the emotional state of staff.

[0437] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0438] In this invention, the server includes recording means for recording user operations, analysis means for analyzing data, and emotion analysis means for recognizing user emotions. This makes it possible to improve work efficiency while reducing stress for care workers.

[0439] A "user" is a person who operates a system and is an executor in a business process—a person who uses an information processing device.

[0440] "Recording means" refers to a device or software that has the function of acquiring user operation information and storing it as data in a format that can be analyzed later.

[0441] "Analysis means" refers to a device or software that analyzes data obtained by recording means and extracts information for subdividing business processes.

[0442] "Decomposition means" refers to methods or apparatus for decomposing a business process into its elements based on the results of analysis performed by analytical means.

[0443] "Detection means" refers to methods or devices for identifying parts of a process that can be made more efficient based on the process that has been disassembled by the disassembly means.

[0444] "Generation means" refers to methods or devices for creating new business workflows based on the areas for improvement identified by the detection means.

[0445] "Design and development means" refers to methods and devices for constructing a support system to provide work assistance based on the generated business flow.

[0446] "Distribution means" refers to methods or devices for providing support systems developed through design and development means to each user's information processing device.

[0447] "Emotional analysis means" refers to a device or software that has the function of identifying a user's emotional state and converting that information into a format that can be used to improve work efficiency.

[0448] "Emotion-based metrics" are data obtained by quantifying or evaluating user emotional information acquired through emotion analysis methods, and are used to improve work efficiency.

[0449] This invention is a system aimed at streamlining the work of staff in caregiving settings while simultaneously reducing their mental burden. This system uses smart glasses, which are information processing devices worn by the user. The device is equipped with recording means for recording the user's operations and actions. The recording means has the function of acquiring the staff member's voice, movements, and related biometric data.

[0450] The device transmits recorded data to the server via a secure protocol. The data is compressed and encrypted in real time, ensuring security.

[0451] The server leverages AWS cloud services to efficiently analyze large amounts of data. It uses multiple data analysis algorithms written in Python. The server breaks down the transmitted data into multiple elements and also analyzes the user's emotional state. Google's Sentiment Analysis API is used for this sentiment analysis.

[0452] Based on the analysis results, the server designs and develops a work support system. The interactive work support system is designed to assist the user based on the generated workflow and emotional state. Depending on the design and development method, it may be possible to prompt users experiencing high stress levels to take short breaks.

[0453] Ultimately, the support system is distributed to the device and assists the user's work processes. For example, if emotion analysis indicates that a staff member working in a nursing home is experiencing increased stress during routine tasks, they will receive a notification prompting them to take a break at an appropriate time. This is expected to improve the quality of care and reduce the workload on staff.

[0454] An example of a prompt to input into the generating AI model would be: "We are developing a system to streamline users' business processes. The system will recognize users' emotions and use that information to optimize operations. We are seeking specific application proposals that take into account applications in nursing care settings."

[0455] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0456] Step 1:

[0457] When a user begins work, the device's recording capabilities acquire user actions, voice, movements, and biometric data such as heart rate in real time. Input consists of user actions and biometric information, while output is compressed data of this information. Data collection is performed using sensors within the smart glasses.

[0458] Step 2:

[0459] The terminal compresses the recorded data in real time and sends it to the server using a secure protocol. The input is the raw data before compression, and the output is encrypted data sent to the server. Here, preparations are made to securely send the data to the server using encryption algorithms such as AES.

[0460] Step 3:

[0461] The server receives the transmitted data and analyzes it using analytical tools. The input is encrypted data, and the output is the analyzed business process data. First, the data is decrypted, and an analysis algorithm using Python is executed to identify the business flow and emotional state.

[0462] Step 4:

[0463] Based on the analysis results, the server uses a decomposition tool to break down the business process into specific elements. The input is the analysis results, and the output is a list of the decomposed business processes. The business process is divided into individual tasks, and the possibility of improvement is evaluated for each task.

[0464] Step 5:

[0465] The server uses a generation method to link decomposed information with sentiment data and design an optimized workflow. The input is metric data obtained from sentiment analysis and information on decomposed business processes, and the output is the optimized new workflow. It processes sentiment data using Google's sentiment analysis API and automatically suggests stress reduction and efficient work assignments.

[0466] Step 6:

[0467] The server develops a work support system aligned with the user's workflow through design and development tools, and distributes it to the terminal. The input is the optimized workflow, and the output is the customized program incorporated into the distributed work support system. Software updates are performed on the terminal, and notifications are configured to be displayed on the smart glasses' display.

[0468] Step 7:

[0469] The user's terminal accepts the distributed work support system and provides support information in real time during work based on that system. Input is data from the distributed support system, and output is notifications and alerts for work improvement displayed to the user. For example, if the stress level is high, a message recommending a 10-minute break will be displayed on the screen.

[0470] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0471] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0472] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0473] [Third Embodiment]

[0474] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0475] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0476] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0477] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0478] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0479] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0480] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0481] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0482] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0483] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0484] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0485] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0486] This invention provides a system for automating user tasks by recording PC screen operations and using AI to analyze and streamline business processes. This system consists of a user, a terminal, and a server.

[0487] 1. User actions

[0488] The user launches a dedicated application and records their screen operations during work. This generates data that includes the user's daily work activities.

[0489] 2. Terminal Processing

[0490] The terminal performs the necessary compression and conversion of the recorded data before sending it to the server. This process can also include encryption to ensure data security.

[0491] 3. Server analysis

[0492] The server analyzes the received data. Here, AI technology is used to identify user actions and break down business processes step by step. The analysis results include the detection of patterns in actions and recurring tasks.

[0493] 4. Generating business workflows

[0494] Based on the analysis results, the server automatically creates the current business process (As-Is flow) and the optimized business process (To-Be flow). This process is presented visually to the user, showing which tasks can be automated.

[0495] 5. Design and Development of AI Agents

[0496] The server designs an AI agent based on the To-Be business flow and automatically generates code. This agent is a program designed to automate the user's business processes and efficiently handle routine tasks.

[0497] 6. Agent Distribution and Implementation

[0498] The generated AI agent is distributed to the device and works in conjunction with the applications the user uses. This agent allows users to reduce the effort required for repetitive tasks, freeing up more time for other creative work.

[0499] Specific example

[0500] For example, for a user who performs data entry tasks daily, this system can record the process of the user manually entering data, and the AI ​​analyzes the entire process. As a result, an automated data entry agent can be designed, enabling daily data entry tasks to be performed with less time and effort.

[0501] As a result, even users without a technical background can easily streamline their work and dramatically improve operational efficiency.

[0502] The following describes the processing flow.

[0503] Step 1:

[0504] The user launches a recording application and records the screen operations of a business process. At this time, the user selects a business process that includes a specific operation and starts recording.

[0505] Step 2:

[0506] The terminal receives the recorded data, compresses it according to the data format, encrypts it as needed, and prepares it for appropriate transfer to the server.

[0507] Step 3:

[0508] The server receives the recorded data sent from the terminal. It decompresses the received data and converts it into a format that can be analyzed by AI.

[0509] Step 4:

[0510] The AI ​​on the server sequentially analyzes the recorded data, classifying and identifying specific user actions such as mouse clicks and keyboard inputs. This allows it to understand the steps that make up the business process.

[0511] Step 5:

[0512] Based on the analysis results, the server breaks down the identified business processes into steps. This breakdown particularly marks repetitive tasks and monotonous work areas.

[0513] Step 6:

[0514] The server automatically generates the current business flow (As-Is) and the optimized business flow (To-Be) based on the broken-down business processes. The generated flows are output in a visual flowchart format that is easy for users to understand.

[0515] Step 7:

[0516] The server designs an AI agent based on the To-Be business flow and generates an automation program to execute it. This program utilizes specific scripts and APIs to automate business processes.

[0517] Step 8:

[0518] The server packages the generated AI agent and performs the distribution procedure to the terminal. This prepares the user's terminal to accept the AI ​​agent.

[0519] Step 9:

[0520] Users run the AI ​​agent on their device and observe how their work processes are automated. If necessary, users can fine-tune the agent's behavior to optimize work efficiency.

[0521] (Example 1)

[0522] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0523] Traditional business automation systems often lack the ability to accurately record and analyze user actions and propose efficient work procedures. Furthermore, most systems require complex configurations, making them difficult for users without a technical background to utilize. There is also a need to effectively identify areas for business efficiency improvements and to rapidly develop and distribute automation programs.

[0524] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0525] In this invention, the server includes an analysis means for analyzing the information and identifying the operation content, a decomposition means for breaking down the work procedure based on the results of the analysis, and a detection means for detecting areas where the operation can be made more efficient based on the procedure. This enables accurate analysis of the user's work procedure, automatic detection of parts that can be made more efficient, and rapid development and distribution of AI automation programs.

[0526] "Recording means" refers to a device or function that electronically captures user behavior and stores that data.

[0527] "Processing means" refers to devices or functions that compress and encrypt the obtained information, enabling secure and efficient data transfer.

[0528] "Transmission means" refers to communication functions or devices used to send processed information to a server.

[0529] "Analysis means" refers to functions or algorithms that analyze information received by the server and identify the user's actions.

[0530] A "disassembly method" refers to a device or function used to break down a work procedure in detail based on the analyzed operation content.

[0531] "Detection means" refers to devices or functions used to identify areas for improvement in business procedures that have been broken down.

[0532] A "generation means" refers to a device or function that creates new work procedures based on the detected areas for improvement.

[0533] "Design and development means" refers to the functions and devices used to design and develop AI automation programs based on generated business procedures.

[0534] "Distribution means" refers to the system or function for providing and installing the developed AI automation program on the user's terminal.

[0535] This invention is a system aimed at automating business processes, providing a series of processes for recording and analyzing user operations and generating automation programs. Specific embodiments are described below.

[0536] Users launch a dedicated application on their work terminal to record their operations during work. During this process, the user's mouse movements, keyboard input, and screen interactions are recorded in detail. For example, it can capture how routine data entry tasks are performed.

[0537] The terminal processes the recorded operation data and compresses and encrypts it to reduce data size and ensure security. This makes it possible to transmit data to the server efficiently and securely. Specifically, the AES encryption method is often used to enhance security.

[0538] Data is sent to a server, which uses an AI model to analyze user actions based on the received data and breaks down the content into step-by-step business procedures. The server utilizes machine learning algorithms to detect repetitive operation patterns and procedures that can be made more efficient. Based on the analysis results, a newly optimized business procedure (To-Be flow) is automatically generated.

[0539] The server designs and develops AI automation programs (agents) based on the generated business procedures. These agents automate the user's business processes and efficiently handle repetitive tasks. The designed and developed programs are distributed to the user's terminal and work in conjunction with the applications the user uses daily. This frees the user from simple data entry tasks, allowing them to dedicate more time to more creative work.

[0540] A concrete example is the automation of data entry tasks that users perform on a daily basis. Using this system, the AI ​​analyzes the input process and generates a prompt message such as, "Please record all the steps necessary for the user to streamline data entry and design a process to automate it," and an automated input agent is developed. By inputting this example prompt message, it becomes easy to analyze and automate other similar business flows.

[0541] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0542] Step 1:

[0543] Users launch a dedicated recording tool within their business applications to record their daily work operations. Specifically, the software captures the user's input device operations (mouse clicks, keyboard input, etc.) and saves them as data in chronological order. This operation history data is the input, and the output is the recorded operation data file.

[0544] Step 2:

[0545] The terminal compresses and encrypts user-generated operation data. In this process, the operation data file is used as input, and the data size is reduced by a compression algorithm. Furthermore, data security is ensured using encryption technologies such as AES. The output is the compressed and encrypted data file.

[0546] Step 3:

[0547] The terminal sends compressed and encrypted data files to the server. Here, a communication protocol (e.g., HTTPS) is used to ensure secure and reliable data transfer. The input is the compressed and encrypted data files, and the output is the data stored on the server.

[0548] Step 4:

[0549] The server decompresses and decrypts the received data, preparing it for analysis. The decompressed and decrypted data is the input, which restores the original operation data, making it possible to analyze it with the AI ​​model. The output is the operation data converted into an analyzable state.

[0550] Step 5:

[0551] The server uses an AI model to analyze operation data and identify user actions. Machine learning algorithms extract operation features and analyze each step to reveal the work procedure. The input is analyzable operation data, and the output is a set of identified operation patterns and steps.

[0552] Step 6:

[0553] The server breaks down business procedures based on the analysis results and detects areas for improvement. It utilizes process analysis tools to identify which parts can be made more efficient. The input is the identified operation patterns, and the output is the broken-down business procedures and areas for improvement.

[0554] Step 7:

[0555] The server generates new business procedures based on improvement points and designs and develops AI automation programs. It generates code in programming languages ​​such as Python and Java for the parts that can be automated. The input is the decomposed business procedures and improvement points, and the output is the designed and developed AI agent.

[0556] Step 8:

[0557] The generated AI agent is distributed to the terminal and runs in conjunction with the user's business application. The user confirms that their daily tasks are automated by the implemented AI agent. The input is the AI ​​agent, and the output is the automated business process.

[0558] (Application Example 1)

[0559] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0560] Traditional factory operations often involve a significant amount of manual labor, leaving many areas unautomated. This results in inefficient work processes and a heavy burden on workers, necessitating increased efficiency and automation of work processes. Furthermore, the higher the level of specialization required for a task, the more difficult it becomes to standardize it, posing a significant hurdle to automation.

[0561] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0562] In this invention, the server includes recording means for recording user operations, communication means for compressing and encrypting the recorded data and transmitting it to the server, and execution means having control functions for automating factory operations. This enables efficient analysis and automation of factory work processes, thereby improving work efficiency and reducing the burden on workers.

[0563] "Means of recording user operations" refer to devices or software used to digitally save user work content and procedures.

[0564] "A communication method for compressing and encrypting recorded data and sending it to a server" refers to a technology for sending collected data to a remote server in a reduced size and with improved security.

[0565] "Execution means with control functions for automating factory work" refers to technology that executes instructions and operations to mechanically replace manual work in a factory's manufacturing process.

[0566] A "server" is a digital device that receives recorded data, analyzes it, and performs the calculations and instructions necessary to optimize the automated process.

[0567] The system for realizing this invention consists of three main elements: a user, a terminal, and a server. The user uses recording means to record their operations. For example, by utilizing PC screen recording software, the user can collect the procedures for their daily work as digital data.

[0568] The terminal receives the recorded data, compresses and encrypts it for communication, and reliably transmits it to the server. The common AES encryption method is often used to ensure data security.

[0569] The server operates in a high-performance computing environment and analyzes the transmitted data using generative AI models. Machine learning frameworks such as TensorFlow and PyTorch are used during the analysis process. The server breaks down the user's operational tasks and identifies areas that can be optimized. These specific operational procedures are then implemented in robots to automate factory operations.

[0570] As a concrete example, consider the parts sorting and assembly processes on a factory's production line. By introducing this system, manual work procedures are recorded, analyzed by a server, and parts that can be automated are designed as AI agents. These agents are then incorporated into robots within the factory, significantly improving work efficiency.

[0571] By utilizing the operation of the generated program, users can focus on more creative tasks. Examples of prompts using the generated AI model include, "Analyze the sorting procedure for this part and design the optimal automation flow," and "Generate a program for the robot to maximize the efficiency of a specific assembly line."

[0572] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0573] Step 1:

[0574] The user records the steps taken during a business process. Specifically, they launch PC screen recording software and capture the operations in digital format. The input is the user's operation information, and the output is the recorded data.

[0575] Step 2:

[0576] The terminal compresses and encrypts the recorded data from the user. The input is the recorded data, and the output is the compressed, encrypted data. Here, the data is made into a size that can be efficiently transmitted, and security is enhanced using AES or similar methods.

[0577] Step 3:

[0578] The terminal sends compressed and encrypted data to the server. The input is compressed and encrypted data, and the output is securely transferred data. The data is transferred to the cloud server using a communication protocol.

[0579] Step 4:

[0580] The server decrypts the received data and prepares it for analysis. The input is compressed encrypted data, and the output is the decrypted data for analysis.

[0581] Step 5:

[0582] The server uses a generated AI model to analyze the user's work process. The input is decoded analysis data, and the output is information on the decomposition and optimization of the business process. Analysis is performed using tools such as TensorFlow and PyTorch.

[0583] Step 6:

[0584] The server designs an optimized business workflow based on the analysis results. The input is decomposition information of the business process, and the output is the To-Be flow. The server visually presents the steps that can be made more efficient.

[0585] Step 7:

[0586] The server automatically generates the design and code for the AI ​​agent based on the To-Be flow. The input is the To-Be flow, and the output is the implementation code for the AI ​​agent. The generated code is specialized for robotics.

[0587] Step 8:

[0588] The server distributes AI agents to the terminals. The input is the implementation code of the AI ​​agent, and the output is the distribution status to the user's device.

[0589] Step 9:

[0590] Users operate robots using the received AI agent to automate factory operations. The input is the AI ​​agent, and the output is the automated factory work. Based on the agent, the robot automatically performs the streamlined tasks.

[0591] In each of the above steps, measures have been taken to ensure that the creation and analysis of prompt sentences using the generative AI model proceeds smoothly.

[0592] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0593] This invention provides a system that combines a system for streamlining user business processes by recording and analyzing PC screen operations with an emotion engine that recognizes user emotions. This system consists of a user, a terminal, and a server, and provides advanced support for automating user tasks.

[0594] 1. User actions

[0595] The user launches a dedicated application to begin their work. The workflow is recorded, and the user's actions are captured in real time. This recording includes emotional information such as the user's voice, facial expressions, and heart rate.

[0596] 2. Terminal Processing

[0597] The device compresses recorded operational and emotional data in real time and securely transmits it to the server. Emotional data can be anonymized to protect individual privacy.

[0598] 3. Server analysis

[0599] The server analyzes the transmitted operation data. This analysis uses AI technology to break down the user's actions into smaller parts. In addition, an emotion engine analyzes the user's emotional state, making it possible to quantify it as stress, satisfaction, and other factors.

[0600] 4. Optimization based on business flow and sentiment analysis

[0601] The server analyzes operational data and incorporates emotional data to consider how to optimize business processes. This allows for optimization that can reduce user psychological stress and improve satisfaction. The generated business flow is adjusted to reflect the user's emotional state.

[0602] 5. Design and Development of AI Agents

[0603] The server designs AI agents based on optimized workflows. These agents can not only automate actual tasks but also adapt their responses to the user's emotions.

[0604] 6. Agent Distribution and Implementation

[0605] The designed AI agent is distributed to the device and operates in the user's environment. The agent is integrated into the user's workflow and provides emotion-based, optimal support.

[0606] Specific example

[0607] For example, in the case of a support staff member who frequently interacts with customers, this system can automatically suggest optimization measures such as reducing workload or sending break notifications when the emotion engine detects that stress levels are high. This improves the ease of work for the staff member and can lead to increased long-term productivity.

[0608] Through the above, this system adds a new element—emotion—to business automation, enabling the proposal of work styles that are more considerate of human psychological states.

[0609] The following describes the processing flow.

[0610] Step 1:

[0611] The user launches a dedicated application and begins their work. The application records the user's screen activity and simultaneously records emotional data using a microphone, camera, and heart rate sensor.

[0612] Step 2:

[0613] The device compresses recorded operation data and emotion data in real time and encrypts the data to ensure security. It then prepares the data for transmission to the server.

[0614] Step 3:

[0615] The server analyzes the received operation data and uses AI to break down the user's work actions. Specifically, it identifies actions such as mouse clicks, keyboard input, and window switching.

[0616] Step 4:

[0617] The server utilizes an emotion engine to analyze the user's emotional state from their facial expressions, voice, heart rate, and other data. Based on these results, it quantifies the user's stress level and fatigue level.

[0618] Step 5:

[0619] The server integrates analysis results of operational data with emotional data to optimize business processes and user psychological states. This includes generating improvement suggestions to adjust work processes when the workload is high or when emotional data indicates high stress levels.

[0620] Step 6:

[0621] The server automatically generates a workflow as the "To-Be" state, taking into account emotions and efficiency. It creates a workflow that considers stress reduction so that users can perform their tasks with peace of mind.

[0622] Step 7:

[0623] The server designs an AI agent based on the newly generated To-Be workflow and creates code to provide the optimal work environment for the user's psychological state. This agent includes scripts that adjust tasks according to the user's state.

[0624] Step 8:

[0625] Agents are distributed to terminals, and users run these agents. Based on sentiment analysis, the agents automatically review task assignments, change task priorities as needed, and notify users.

[0626] Step 9:

[0627] Users can verify that their tasks are automatically adjusted by the AI ​​agent and provide feedback on their actions approximately three times a day (morning, noon, and evening) based on their emotional changes. This feedback allows the agent to learn further and use it to make adjustments in the future.

[0628] (Example 2)

[0629] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0630] Traditional business automation systems focused on efficiency improvements to enhance user productivity, but they lacked sufficient optimization that considered user emotional states. As a result, reducing user psychological stress and improving job satisfaction were not adequately achieved. Therefore, there is a need for systems that not only improve operational efficiency but also optimize by reflecting user emotional states in real time.

[0631] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0632] In this invention, the server includes acquisition means for recording user operations and emotional states, transmission means for compressing and transmitting the data obtained by the acquisition means, and analysis means for analyzing the data transmitted by the transmission means and decomposing work steps and emotional states. This makes it possible to streamline the user's workflow while reducing psychological stress and improving job satisfaction.

[0633] "User actions and emotional state" refers to the state of the user, including both physical actions (keyboard input, mouse operation, etc.) and psychological responses such as heart rate and facial expressions.

[0634] "Acquisition means" refers to mechanical or software devices for collecting and recording user actions and emotional states.

[0635] "Transmission means" refers to a communication function that compresses acquired data into a predetermined format and securely transfers it to a server.

[0636] "Analysis means" refers to an algorithm or device that uses transmitted data to subdivide business processes and evaluate the user's emotional state.

[0637] "Design and generation methods" refer to processes or functions that streamline business workflows and automatically generate AI agents based on data obtained through analysis.

[0638] "Distribution means" refers to the function of providing the designed and generated AI agent to the user's device and ensuring its proper installation and execution.

[0639] A "business process flow" refers to a planned set of steps or a set of procedures designed to optimize a user's business processes.

[0640] This invention is a system for optimizing work processes by recording user actions and emotional states. Users begin their work using a dedicated application. The application, installed on a desktop or laptop computer, captures user actions and emotional states in real time. Specifically, it collects user interactions such as keyboard input and mouse operations, as well as emotional information such as heart rate and facial expressions. This data is acquired through wearable devices and cameras.

[0641] The terminal compresses the collected data and sends it to the server using a secure communication protocol. Commonly used compression algorithms (e.g., H.264 encoding or ZIP compression) are employed for data compression, and SSL / TLS encryption is used for communication.

[0642] The server analyzes the transmitted data using advanced AI algorithms. This analysis breaks down business processes and identifies common work patterns. It also quantifies users' psychological states from emotional data and evaluates stress and satisfaction levels. A cloud-based AI platform is used for the analysis.

[0643] The data analyzed on the server is used to optimize business workflows. This optimization includes adjustments to reduce user stress and suggestions for more efficient work sequences. This enables the creation of a more comfortable and productive work environment.

[0644] As a concrete example, when a support staff member who frequently interacts with customers uses this system, if the emotion engine determines that the user is experiencing high stress levels, it will automatically suggest taking a break and provide other support. The AI ​​agent generated for this purpose is designed and developed in Python or Java and distributed to the user's device.

[0645] An example of a prompt to input into a generative AI model might be, "Explain how to use an emotion engine to detect stress levels in a sales team and design automated responses." Such prompts allow the system to automatically analyze the user's work and derive solutions.

[0646] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0647] Step 1:

[0648] The user launches a dedicated application to begin work. The application acquires the user's keyboard input and mouse operations, as well as heart rate and facial recognition camera data received from connected wearable devices, in real time. Input data includes keyboard and mouse event logs, heart rate, and facial expression data. This data is output as a log organized by time and operation using the application's built-in functions.

[0649] Step 2:

[0650] The terminal compresses the log data acquired in Step 1 and prepares it for efficient transfer to the server. Specifically, it encodes image data in JPEG format, video data in H.264 format, and compiles event logs in JSON format. These compressed datasets are encrypted using SSL / TLS and securely sent to the server. The input is the log data obtained in Step 1, and the output is the compressed and encrypted data for transmission.

[0651] Step 3:

[0652] The server receives data transmitted from the terminal and performs analysis using an AI algorithm. During data analysis, it thoroughly analyzes the operation details and models the workflow. Furthermore, an emotion engine quantifies the user's emotional state from heart rate and facial expression data. At this stage, the input is compressed and encrypted transmitted data, which is then decompressed and analyzed to output the breakdown of business steps and numerical values ​​representing the emotional state.

[0653] Step 4:

[0654] The server optimizes the business workflow based on the analysis results. It identifies steps where efficiency can be improved according to predefined criteria and constructs a new business process that takes into account the user's emotional state. In this process, the input consists of the business step data and quantified emotional information obtained in step 3, while the output is a model of the optimized business workflow.

[0655] Step 5:

[0656] Based on the optimized workflow, the server automatically designs and generates an AI agent. The AI ​​agent is coded using a programming language and tested and verified on the server side. The input is the workflow obtained in step 4, and the output is the code for a workable AI agent.

[0657] Step 6:

[0658] The server distributes the designed and generated AI agent to the terminal and deploys it into the user's work environment. The agent is integrated into the system and provides the user with timely, emotion-based support. The input is the AI ​​agent code obtained in step 5, and the output is the operational agent implemented on the user's terminal.

[0659] (Application Example 2)

[0660] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0661] In the field of elderly care, staff workloads and stress levels are often increasing, leading to challenges such as decreased work efficiency and a decline in the quality of care. To address this, there is a need for systems that streamline work processes while considering the emotional state of staff.

[0662] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0663] In this invention, the server includes recording means for recording user operations, analysis means for analyzing data, and emotion analysis means for recognizing user emotions. This makes it possible to improve work efficiency while reducing stress for care workers.

[0664] A "user" is a person who operates a system and is an executor in a business process—a person who uses an information processing device.

[0665] "Recording means" refers to a device or software that has the function of acquiring user operation information and storing it as data in a format that can be analyzed later.

[0666] "Analysis means" refers to a device or software that analyzes data obtained by recording means and extracts information for subdividing business processes.

[0667] "Decomposition means" refers to methods or apparatus for decomposing a business process into its elements based on the results of analysis performed by analytical means.

[0668] "Detection means" refers to methods or devices for identifying parts of a process that can be made more efficient based on the process that has been disassembled by the disassembly means.

[0669] "Generation means" refers to methods or devices for creating new business workflows based on the areas for improvement identified by the detection means.

[0670] "Design and development means" refers to methods and devices for constructing a support system to provide work assistance based on the generated business flow.

[0671] "Distribution means" refers to methods or devices for providing support systems developed through design and development means to each user's information processing device.

[0672] "Emotional analysis means" refers to a device or software that has the function of identifying a user's emotional state and converting that information into a format that can be used to improve work efficiency.

[0673] "Emotion-based metrics" are data obtained by quantifying or evaluating user emotional information acquired through emotion analysis methods, and are used to improve work efficiency.

[0674] This invention is a system aimed at streamlining the work of staff in caregiving settings while simultaneously reducing their mental burden. This system uses smart glasses, which are information processing devices worn by the user. The device is equipped with recording means for recording the user's operations and actions. The recording means has the function of acquiring the staff member's voice, movements, and related biometric data.

[0675] The device transmits recorded data to the server via a secure protocol. The data is compressed and encrypted in real time, ensuring security.

[0676] The server leverages AWS cloud services to efficiently analyze large amounts of data. It uses multiple data analysis algorithms written in Python. The server breaks down the transmitted data into multiple elements and also analyzes the user's emotional state. Google's Sentiment Analysis API is used for this sentiment analysis.

[0677] Based on the analysis results, the server designs and develops a work support system. The interactive work support system is designed to assist the user based on the generated workflow and emotional state. Depending on the design and development method, it may be possible to prompt users experiencing high stress levels to take short breaks.

[0678] Ultimately, the support system is distributed to the device and assists the user's work processes. For example, if emotion analysis indicates that a staff member working in a nursing home is experiencing increased stress during routine tasks, they will receive a notification prompting them to take a break at an appropriate time. This is expected to improve the quality of care and reduce the workload on staff.

[0679] An example of a prompt to input into the generating AI model would be: "We are developing a system to streamline users' business processes. The system will recognize users' emotions and use that information to optimize operations. We are seeking specific application proposals that take into account applications in nursing care settings."

[0680] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0681] Step 1:

[0682] When a user begins work, the device's recording capabilities acquire user actions, voice, movements, and biometric data such as heart rate in real time. Input consists of user actions and biometric information, while output is compressed data of this information. Data collection is performed using sensors within the smart glasses.

[0683] Step 2:

[0684] The terminal compresses the recorded data in real time and sends it to the server using a secure protocol. The input is the raw data before compression, and the output is encrypted data sent to the server. Here, preparations are made to securely send the data to the server using encryption algorithms such as AES.

[0685] Step 3:

[0686] The server receives the transmitted data and analyzes it using analytical tools. The input is encrypted data, and the output is the analyzed business process data. First, the data is decrypted, and an analysis algorithm using Python is executed to identify the business flow and emotional state.

[0687] Step 4:

[0688] Based on the analysis results, the server uses a decomposition tool to break down the business process into specific elements. The input is the analysis results, and the output is a list of the decomposed business processes. The business process is divided into individual tasks, and the possibility of improvement is evaluated for each task.

[0689] Step 5:

[0690] The server uses a generation method to link decomposed information with sentiment data and design an optimized workflow. The input is metric data obtained from sentiment analysis and information on decomposed business processes, and the output is the optimized new workflow. It processes sentiment data using Google's sentiment analysis API and automatically suggests stress reduction and efficient work assignments.

[0691] Step 6:

[0692] The server develops a work support system aligned with the user's workflow through design and development tools, and distributes it to the terminal. The input is the optimized workflow, and the output is the customized program incorporated into the distributed work support system. Software updates are performed on the terminal, and notifications are configured to be displayed on the smart glasses' display.

[0693] Step 7:

[0694] The user's terminal accepts the distributed work support system and provides support information in real time during work based on that system. Input is data from the distributed support system, and output is notifications and alerts for work improvement displayed to the user. For example, if the stress level is high, a message recommending a 10-minute break will be displayed on the screen.

[0695] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0696] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0697] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0698] [Fourth Embodiment]

[0699] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0700] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0701] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0702] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0703] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0704] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0705] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0706] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0707] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0708] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0709] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0710] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0711] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0712] This invention provides a system for automating user tasks by recording PC screen operations and using AI to analyze and streamline business processes. This system consists of a user, a terminal, and a server.

[0713] 1. User actions

[0714] The user launches a dedicated application and records their screen operations during work. This generates data that includes the user's daily work activities.

[0715] 2. Terminal Processing

[0716] The terminal performs the necessary compression and conversion of the recorded data before sending it to the server. This process can also include encryption to ensure data security.

[0717] 3. Server analysis

[0718] The server analyzes the received data. Here, AI technology is used to identify user actions and break down business processes step by step. The analysis results include the detection of patterns in actions and recurring tasks.

[0719] 4. Generating business workflows

[0720] Based on the analysis results, the server automatically creates the current business process (As-Is flow) and the optimized business process (To-Be flow). This process is presented visually to the user, showing which tasks can be automated.

[0721] 5. Design and Development of AI Agents

[0722] The server designs an AI agent based on the To-Be business flow and automatically generates code. This agent is a program designed to automate the user's business processes and efficiently handle routine tasks.

[0723] 6. Agent Distribution and Implementation

[0724] The generated AI agent is distributed to the device and works in conjunction with the applications the user uses. This agent allows users to reduce the effort required for repetitive tasks, freeing up more time for other creative work.

[0725] Specific example

[0726] For example, for a user who performs data entry tasks daily, this system can record the process of the user manually entering data, and the AI ​​analyzes the entire process. As a result, an automated data entry agent can be designed, enabling daily data entry tasks to be performed with less time and effort.

[0727] As a result, even users without a technical background can easily streamline their work and dramatically improve operational efficiency.

[0728] The following describes the processing flow.

[0729] Step 1:

[0730] The user launches a recording application and records the screen operations of a business process. At this time, the user selects a business process that includes a specific operation and starts recording.

[0731] Step 2:

[0732] The terminal receives the recorded data, compresses it according to the data format, encrypts it as needed, and prepares it for appropriate transfer to the server.

[0733] Step 3:

[0734] The server receives the recorded data sent from the terminal. It decompresses the received data and converts it into a format that can be analyzed by AI.

[0735] Step 4:

[0736] The AI ​​on the server sequentially analyzes the recorded data, classifying and identifying specific user actions such as mouse clicks and keyboard inputs. This allows it to understand the steps that make up the business process.

[0737] Step 5:

[0738] Based on the analysis results, the server breaks down the identified business processes into steps. This breakdown particularly marks repetitive tasks and monotonous work areas.

[0739] Step 6:

[0740] The server automatically generates the current business flow (As-Is) and the optimized business flow (To-Be) based on the broken-down business processes. The generated flows are output in a visual flowchart format that is easy for users to understand.

[0741] Step 7:

[0742] The server designs an AI agent based on the To-Be business flow and generates an automation program to execute it. This program utilizes specific scripts and APIs to automate business processes.

[0743] Step 8:

[0744] The server packages the generated AI agent and performs the distribution procedure to the terminal. This prepares the user's terminal to accept the AI ​​agent.

[0745] Step 9:

[0746] Users run the AI ​​agent on their device and observe how their work processes are automated. If necessary, users can fine-tune the agent's behavior to optimize work efficiency.

[0747] (Example 1)

[0748] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0749] Traditional business automation systems often lack the ability to accurately record and analyze user actions and propose efficient work procedures. Furthermore, most systems require complex configurations, making them difficult for users without a technical background to utilize. There is also a need to effectively identify areas for business efficiency improvements and to rapidly develop and distribute automation programs.

[0750] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0751] In this invention, the server includes an analysis means for analyzing the information and identifying the operation content, a decomposition means for breaking down the work procedure based on the results of the analysis, and a detection means for detecting areas where the operation can be made more efficient based on the procedure. This enables accurate analysis of the user's work procedure, automatic detection of parts that can be made more efficient, and rapid development and distribution of AI automation programs.

[0752] "Recording means" refers to a device or function that electronically captures user behavior and stores that data.

[0753] "Processing means" refers to devices or functions that compress and encrypt the obtained information, enabling secure and efficient data transfer.

[0754] "Transmission means" refers to communication functions or devices used to send processed information to a server.

[0755] "Analysis means" refers to functions or algorithms that analyze information received by the server and identify the user's actions.

[0756] A "disassembly means" refers to a device or function used to break down a work procedure in detail based on the analyzed operation content.

[0757] "Detection means" refers to devices or functions used to identify areas for improvement in business procedures that have been broken down.

[0758] A "generation means" refers to a device or function that creates new work procedures based on the detected areas for improvement.

[0759] "Design and development means" refers to the functions and devices used to design and develop AI automation programs based on generated business procedures.

[0760] "Distribution means" refers to the system or function for providing and installing the developed AI automation program on the user's terminal.

[0761] This invention is a system aimed at automating business processes, providing a series of processes for recording and analyzing user operations and generating automation programs. Specific embodiments are described below.

[0762] Users launch a dedicated application on their work terminal to record their operations during work. During this process, the user's mouse movements, keyboard input, and screen interactions are recorded in detail. For example, it can capture how routine data entry tasks are performed.

[0763] The terminal processes the recorded operation data and compresses and encrypts it to reduce data size and ensure security. This makes it possible to transmit data to the server efficiently and securely. Specifically, the AES encryption method is often used to enhance security.

[0764] Data is sent to a server, which uses an AI model to analyze user actions based on the received data and breaks down the content into step-by-step business procedures. The server utilizes machine learning algorithms to detect repetitive operation patterns and procedures that can be made more efficient. Based on the analysis results, a newly optimized business procedure (To-Be flow) is automatically generated.

[0765] The server designs and develops AI automation programs (agents) based on the generated business procedures. These agents automate the user's business processes and efficiently handle repetitive tasks. The designed and developed programs are distributed to the user's terminal and work in conjunction with the applications the user uses daily. This frees the user from simple data entry tasks, allowing them to dedicate more time to more creative work.

[0766] A concrete example is the automation of data entry tasks that users perform on a daily basis. Using this system, the AI ​​analyzes the input process and generates a prompt message such as, "Please record all the steps necessary for the user to streamline data entry and design a process to automate it," and an automated input agent is developed. By inputting this example prompt message, it becomes easy to analyze and automate other similar business flows.

[0767] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0768] Step 1:

[0769] Users launch a dedicated recording tool within their business applications to record their daily work operations. Specifically, the software captures the user's input device operations (mouse clicks, keyboard input, etc.) and saves them as data in chronological order. This operation history data is the input, and the output is the recorded operation data file.

[0770] Step 2:

[0771] The terminal compresses and encrypts user-generated operation data. In this process, the operation data file is used as input, and the data size is reduced by a compression algorithm. Furthermore, data security is ensured using encryption technologies such as AES. The output is the compressed and encrypted data file.

[0772] Step 3:

[0773] The terminal sends compressed and encrypted data files to the server. Here, a communication protocol (e.g., HTTPS) is used to ensure secure and reliable data transfer. The input is the compressed and encrypted data files, and the output is the data stored on the server.

[0774] Step 4:

[0775] The server decompresses and decrypts the received data, preparing it for analysis. The decompressed and decrypted data is the input, which restores the original operation data, making it possible to analyze it with the AI ​​model. The output is the operation data converted into an analyzable state.

[0776] Step 5:

[0777] The server uses an AI model to analyze operation data and identify user actions. Machine learning algorithms extract operation features and analyze each step to reveal the work procedure. The input is analyzable operation data, and the output is a set of identified operation patterns and steps.

[0778] Step 6:

[0779] The server breaks down business procedures based on the analysis results and detects areas for improvement. It utilizes process analysis tools to identify which parts can be made more efficient. The input is the identified operation patterns, and the output is the broken-down business procedures and areas for improvement.

[0780] Step 7:

[0781] The server generates new business procedures based on improvement points and designs and develops AI automation programs. It generates code in programming languages ​​such as Python and Java for the parts that can be automated. The input is the decomposed business procedures and improvement points, and the output is the designed and developed AI agent.

[0782] Step 8:

[0783] The generated AI agent is distributed to the terminal and runs in conjunction with the user's business application. The user confirms that their daily tasks are automated by the implemented AI agent. The input is the AI ​​agent, and the output is the automated business process.

[0784] (Application Example 1)

[0785] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0786] Traditional factory operations often involve a significant amount of manual labor, leaving many areas unautomated. This results in inefficient work processes and a heavy burden on workers, necessitating increased efficiency and automation of work processes. Furthermore, the higher the level of specialization required for a task, the more difficult it becomes to standardize it, posing a significant hurdle to automation.

[0787] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0788] In this invention, the server includes recording means for recording user operations, communication means for compressing and encrypting the recorded data and transmitting it to the server, and execution means having control functions for automating factory operations. This enables efficient analysis and automation of factory work processes, thereby improving work efficiency and reducing the burden on workers.

[0789] "Means of recording user operations" refer to devices or software used to digitally save user work content and procedures.

[0790] "A communication method for compressing and encrypting recorded data and sending it to a server" refers to a technology for sending collected data to a remote server in a reduced size and with improved security.

[0791] "Execution means with control functions for automating factory work" refers to technology that executes instructions and operations to mechanically replace manual work in a factory's manufacturing process.

[0792] A "server" is a digital device that receives recorded data, analyzes it, and performs the calculations and instructions necessary to optimize the automated process.

[0793] The system for realizing this invention consists of three main elements: a user, a terminal, and a server. The user uses recording means to record their operations. For example, by utilizing PC screen recording software, the user can collect the procedures for their daily work as digital data.

[0794] The terminal receives the recorded data, compresses and encrypts it for communication, and reliably transmits it to the server. The common AES encryption method is often used to ensure data security.

[0795] The server operates in a high-performance computing environment and analyzes the transmitted data using generative AI models. Machine learning frameworks such as TensorFlow and PyTorch are used during the analysis process. The server breaks down the user's operational tasks and identifies areas that can be optimized. These specific operational procedures are then implemented in robots to automate factory operations.

[0796] As a concrete example, consider the parts sorting and assembly processes on a factory's production line. By introducing this system, manual work procedures are recorded, analyzed by a server, and parts that can be automated are designed as AI agents. These agents are then incorporated into robots within the factory, significantly improving work efficiency.

[0797] By utilizing the operation of the generated program, users can focus on more creative tasks. Examples of prompts using the generated AI model include, "Analyze the sorting procedure for this part and design the optimal automation flow," and "Generate a program for the robot to maximize the efficiency of a specific assembly line."

[0798] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0799] Step 1:

[0800] The user records the steps taken during a business process. Specifically, they launch PC screen recording software and capture the operations in digital format. The input is the user's operation information, and the output is the recorded data.

[0801] Step 2:

[0802] The terminal compresses and encrypts the recorded data from the user. The input is the recorded data, and the output is the compressed, encrypted data. Here, the data is made into a size that can be efficiently transmitted, and security is enhanced using AES or similar methods.

[0803] Step 3:

[0804] The terminal sends compressed and encrypted data to the server. The input is compressed and encrypted data, and the output is securely transferred data. The data is transferred to the cloud server using a communication protocol.

[0805] Step 4:

[0806] The server decrypts the received data and prepares it for analysis. The input is compressed encrypted data, and the output is the decrypted data for analysis.

[0807] Step 5:

[0808] The server uses a generated AI model to analyze the user's work process. The input is decoded analysis data, and the output is information on the decomposition and optimization of the business process. Analysis is performed using tools such as TensorFlow and PyTorch.

[0809] Step 6:

[0810] The server designs an optimized business workflow based on the analysis results. The input is decomposition information of the business process, and the output is the To-Be flow. The server visually presents the steps that can be made more efficient.

[0811] Step 7:

[0812] The server automatically generates the design and code for the AI ​​agent based on the To-Be flow. The input is the To-Be flow, and the output is the implementation code for the AI ​​agent. The generated code is specialized for robotics.

[0813] Step 8:

[0814] The server distributes AI agents to the terminals. The input is the implementation code of the AI ​​agent, and the output is the distribution status to the user's device.

[0815] Step 9:

[0816] Users operate robots using the received AI agent to automate factory operations. The input is the AI ​​agent, and the output is the automated factory work. Based on the agent, the robot automatically performs the streamlined tasks.

[0817] In each of the above steps, measures have been taken to ensure that the creation and analysis of prompt sentences using the generative AI model proceeds smoothly.

[0818] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0819] This invention provides a system that combines a system for streamlining user business processes by recording and analyzing PC screen operations with an emotion engine that recognizes user emotions. This system consists of a user, a terminal, and a server, and provides advanced support for automating user tasks.

[0820] 1. User actions

[0821] The user launches a dedicated application to begin their work. The workflow is recorded, and the user's actions are captured in real time. This recording includes emotional information such as the user's voice, facial expressions, and heart rate.

[0822] 2. Terminal Processing

[0823] The device compresses recorded operational and emotional data in real time and securely transmits it to the server. Emotional data can be anonymized to protect individual privacy.

[0824] 3. Server analysis

[0825] The server analyzes the transmitted operation data. This analysis uses AI technology to break down the user's actions into smaller parts. In addition, an emotion engine analyzes the user's emotional state, making it possible to quantify it as stress, satisfaction, and other factors.

[0826] 4. Optimization based on business flow and sentiment analysis

[0827] The server analyzes operational data and incorporates emotional data to consider how to optimize business processes. This allows for optimization that can reduce user psychological stress and improve satisfaction. The generated business flow is adjusted to reflect the user's emotional state.

[0828] 5. Design and Development of AI Agents

[0829] The server designs AI agents based on optimized workflows. These agents can not only automate actual tasks but also adapt their responses to the user's emotions.

[0830] 6. Agent Distribution and Implementation

[0831] The designed AI agent is distributed to the device and operates in the user's environment. The agent is integrated into the user's workflow and provides emotion-based, optimal support.

[0832] Specific example

[0833] For example, in the case of a support staff member who frequently interacts with customers, this system can automatically suggest optimization measures such as reducing workload or sending break notifications when the emotion engine detects that stress levels are high. This improves the ease of work for the staff member and can lead to increased long-term productivity.

[0834] Through the above, this system adds a new element—emotion—to business automation, enabling the proposal of work styles that are more considerate of human psychological states.

[0835] The following describes the processing flow.

[0836] Step 1:

[0837] The user launches a dedicated application and begins their work. The application records the user's screen activity and simultaneously records emotional data using a microphone, camera, and heart rate sensor.

[0838] Step 2:

[0839] The device compresses recorded operation data and emotion data in real time and encrypts the data to ensure security. It then prepares the data for transmission to the server.

[0840] Step 3:

[0841] The server analyzes the received operation data and uses AI to break down the user's work actions. Specifically, it identifies actions such as mouse clicks, keyboard input, and window switching.

[0842] Step 4:

[0843] The server utilizes an emotion engine to analyze the user's emotional state from their facial expressions, voice, heart rate, and other data. Based on these results, it quantifies the user's stress level and fatigue level.

[0844] Step 5:

[0845] The server integrates analysis results of operational data with emotional data to optimize business processes and user psychological states. This includes generating improvement suggestions to adjust work processes when the workload is high or when emotional data indicates high stress levels.

[0846] Step 6:

[0847] The server automatically generates a workflow as the "To-Be" state, taking into account emotions and efficiency. It creates a workflow that considers stress reduction so that users can perform their tasks with peace of mind.

[0848] Step 7:

[0849] The server designs an AI agent based on the newly generated To-Be workflow and creates code to provide the optimal work environment for the user's psychological state. This agent includes scripts that adjust tasks according to the user's state.

[0850] Step 8:

[0851] Agents are distributed to terminals, and users run these agents. Based on sentiment analysis, the agents automatically review task assignments, change task priorities as needed, and notify users.

[0852] Step 9:

[0853] Users can verify that their tasks are automatically adjusted by the AI ​​agent and provide feedback on their actions approximately three times a day (morning, noon, and evening) based on their emotional changes. This feedback allows the agent to learn further and use it to make adjustments in the future.

[0854] (Example 2)

[0855] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0856] Traditional business automation systems focused on efficiency improvements to enhance user productivity, but they lacked sufficient optimization that considered user emotional states. As a result, reducing user psychological stress and improving job satisfaction were not adequately achieved. Therefore, there is a need for systems that not only improve operational efficiency but also optimize by reflecting user emotional states in real time.

[0857] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0858] In this invention, the server includes acquisition means for recording user operations and emotional states, transmission means for compressing and transmitting the data obtained by the acquisition means, and analysis means for analyzing the data transmitted by the transmission means and decomposing work steps and emotional states. This makes it possible to streamline the user's workflow while reducing psychological stress and improving job satisfaction.

[0859] "User actions and emotional state" refers to the state of the user, including both physical actions (keyboard input, mouse operation, etc.) and psychological responses such as heart rate and facial expressions.

[0860] "Acquisition means" refers to mechanical or software devices for collecting and recording user actions and emotional states.

[0861] "Transmission means" refers to a communication function that compresses acquired data into a predetermined format and securely transfers it to a server.

[0862] "Analysis means" refers to an algorithm or device that uses transmitted data to subdivide business processes and evaluate the user's emotional state.

[0863] "Design and generation methods" refer to processes or functions that streamline business workflows and automatically generate AI agents based on data obtained through analysis.

[0864] "Distribution means" refers to the function of providing the designed and generated AI agent to the user's device and ensuring its proper installation and execution.

[0865] A "business process flow" refers to a planned set of steps or a set of procedures designed to optimize a user's business processes.

[0866] This invention is a system for optimizing work processes by recording user actions and emotional states. Users begin their work using a dedicated application. The application, installed on a desktop or laptop computer, captures user actions and emotional states in real time. Specifically, it collects user interactions such as keyboard input and mouse operations, as well as emotional information such as heart rate and facial expressions. This data is acquired through wearable devices and cameras.

[0867] The terminal compresses the collected data and sends it to the server using a secure communication protocol. Commonly used compression algorithms (e.g., H.264 encoding or ZIP compression) are employed for data compression, and SSL / TLS encryption is used for communication.

[0868] The server analyzes the transmitted data using advanced AI algorithms. This analysis breaks down business processes and identifies common work patterns. It also quantifies users' psychological states from emotional data and evaluates stress and satisfaction levels. A cloud-based AI platform is used for the analysis.

[0869] The data analyzed on the server is used to optimize business workflows. This optimization includes adjustments to reduce user stress and suggestions for more efficient work sequences. This enables the creation of a more comfortable and productive work environment.

[0870] As a concrete example, when a support staff member who frequently interacts with customers uses this system, if the emotion engine determines that the user is experiencing high stress levels, it will automatically suggest taking a break and provide other support. The AI ​​agent generated for this purpose is designed and developed in Python or Java and distributed to the user's device.

[0871] An example of a prompt to input into a generative AI model might be, "Explain how to use an emotion engine to detect stress levels in a sales team and design automated responses." Such prompts allow the system to automatically analyze the user's work and derive solutions.

[0872] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0873] Step 1:

[0874] The user launches a dedicated application to begin work. The application acquires the user's keyboard input and mouse operations, as well as heart rate and facial recognition camera data received from connected wearable devices, in real time. Input data includes keyboard and mouse event logs, heart rate, and facial expression data. This data is output as a log organized by time and operation using the application's built-in functions.

[0875] Step 2:

[0876] The terminal compresses the log data acquired in Step 1 and prepares it for efficient transfer to the server. Specifically, it encodes image data in JPEG format, video data in H.264 format, and compiles event logs in JSON format. These compressed datasets are encrypted using SSL / TLS and securely sent to the server. The input is the log data obtained in Step 1, and the output is the compressed and encrypted data for transmission.

[0877] Step 3:

[0878] The server receives data transmitted from the terminal and performs analysis using an AI algorithm. During data analysis, it thoroughly analyzes the operation details and models the workflow. Furthermore, an emotion engine quantifies the user's emotional state from heart rate and facial expression data. At this stage, the input is compressed and encrypted transmitted data, which is then decompressed and analyzed to output the breakdown of business steps and numerical values ​​representing the emotional state.

[0879] Step 4:

[0880] The server optimizes the business workflow based on the analysis results. It identifies steps where efficiency can be improved according to predefined criteria and constructs a new business process that takes into account the user's emotional state. In this process, the input consists of the business step data and quantified emotional information obtained in step 3, while the output is a model of the optimized business workflow.

[0881] Step 5:

[0882] Based on the optimized workflow, the server automatically designs and generates an AI agent. The AI ​​agent is coded using a programming language and tested and verified on the server side. The input is the workflow obtained in step 4, and the output is the code for a workable AI agent.

[0883] Step 6:

[0884] The server distributes the designed and generated AI agent to the terminal and deploys it into the user's work environment. The agent is integrated into the system and provides the user with timely, emotion-based support. The input is the AI ​​agent code obtained in step 5, and the output is the operational agent implemented on the user's terminal.

[0885] (Application Example 2)

[0886] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0887] In the field of elderly care, staff workloads and stress levels are often increasing, leading to challenges such as decreased work efficiency and a decline in the quality of care. To address this, there is a need for systems that streamline work processes while considering the emotional state of staff.

[0888] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0889] In this invention, the server includes recording means for recording user operations, analysis means for analyzing data, and emotion analysis means for recognizing user emotions. This makes it possible to improve work efficiency while reducing stress for care workers.

[0890] A "user" is a person who operates a system and is an executor in a business process—a person who uses an information processing device.

[0891] "Recording means" refers to a device or software that has the function of acquiring user operation information and storing it as data in a format that can be analyzed later.

[0892] "Analysis means" refers to a device or software that analyzes data obtained by recording means and extracts information for subdividing business processes.

[0893] "Decomposition means" refers to methods or apparatus for decomposing a business process into its elements based on the results of analysis performed by analytical means.

[0894] "Detection means" refers to methods or devices for identifying parts of a process that can be made more efficient based on the process that has been disassembled by the disassembly means.

[0895] "Generation means" refers to methods or devices for creating new business workflows based on the areas for improvement identified by the detection means.

[0896] "Design and development means" refers to methods and devices for constructing a support system to provide work assistance based on the generated business flow.

[0897] "Distribution means" refers to methods or devices for providing support systems developed through design and development means to each user's information processing device.

[0898] "Emotional analysis means" refers to a device or software that has the function of identifying a user's emotional state and converting that information into a format that can be used to improve work efficiency.

[0899] "Emotion-based metrics" are data obtained by quantifying or evaluating user emotional information acquired through emotion analysis methods, and are used to improve work efficiency.

[0900] This invention is a system aimed at streamlining the work of staff in caregiving settings while simultaneously reducing their mental burden. This system uses smart glasses, which are information processing devices worn by the user. The device is equipped with recording means for recording the user's operations and actions. The recording means has the function of acquiring the staff member's voice, movements, and related biometric data.

[0901] The device transmits recorded data to the server via a secure protocol. The data is compressed and encrypted in real time, ensuring security.

[0902] The server leverages AWS cloud services to efficiently analyze large amounts of data. It uses multiple data analysis algorithms written in Python. The server breaks down the transmitted data into multiple elements and also analyzes the user's emotional state. Google's Sentiment Analysis API is used for this sentiment analysis.

[0903] Based on the analysis results, the server designs and develops a work support system. The interactive work support system is designed to assist the user based on the generated workflow and emotional state. Depending on the design and development method, it may be possible to prompt users experiencing high stress levels to take short breaks.

[0904] Ultimately, the support system is distributed to the device and assists the user's work processes. For example, if emotion analysis indicates that a staff member working in a nursing home is experiencing increased stress during routine tasks, they will receive a notification prompting them to take a break at an appropriate time. This is expected to improve the quality of care and reduce the workload on staff.

[0905] An example of a prompt to input into the generating AI model would be: "We are developing a system to streamline users' business processes. The system will recognize users' emotions and use that information to optimize operations. We are seeking specific application proposals that take into account applications in nursing care settings."

[0906] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0907] Step 1:

[0908] When a user begins work, the device's recording capabilities acquire user actions, voice, movements, and biometric data such as heart rate in real time. Input consists of user actions and biometric information, while output is compressed data of this information. Data collection is performed using sensors within the smart glasses.

[0909] Step 2:

[0910] The terminal compresses the recorded data in real time and sends it to the server using a secure protocol. The input is the raw data before compression, and the output is encrypted data sent to the server. Here, preparations are made to securely send the data to the server using encryption algorithms such as AES.

[0911] Step 3:

[0912] The server receives the transmitted data and analyzes it using analytical tools. The input is encrypted data, and the output is the analyzed business process data. First, the data is decrypted, and an analysis algorithm using Python is executed to identify the business flow and emotional state.

[0913] Step 4:

[0914] Based on the analysis results, the server uses a decomposition tool to break down the business process into specific elements. The input is the analysis results, and the output is a list of the decomposed business processes. The business process is divided into individual tasks, and the possibility of improvement is evaluated for each task.

[0915] Step 5:

[0916] The server uses a generation method to link decomposed information with sentiment data and design an optimized workflow. The input is metric data obtained from sentiment analysis and information on decomposed business processes, and the output is the optimized new workflow. It processes sentiment data using Google's sentiment analysis API and automatically suggests stress reduction and efficient work assignments.

[0917] Step 6:

[0918] The server develops a work support system aligned with the user's workflow through design and development tools, and distributes it to the terminal. The input is the optimized workflow, and the output is the customized program incorporated into the distributed work support system. Software updates are performed on the terminal, and notifications are configured to be displayed on the smart glasses' display.

[0919] Step 7:

[0920] The user's terminal accepts the distributed work support system and provides support information in real time during work based on that system. Input is data from the distributed support system, and output is notifications and alerts for work improvement displayed to the user. For example, if the stress level is high, a message recommending a 10-minute break will be displayed on the screen.

[0921] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0922] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0923] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0924] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0925] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0926] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0927] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0928] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0929] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0930] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0931] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0932] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0933] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0934] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0935] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0936] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0937] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0938] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0939] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0940] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0941] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0942] The following is further disclosed regarding the embodiments described above.

[0943] (Claim 1)

[0944] A recording means for recording user operations,

[0945] An analysis means for analyzing the data obtained by the recording means,

[0946] The analysis means includes a decomposition means that decomposes the business process based on the results of the analysis,

[0947] A detection means for detecting areas for improving operational efficiency based on the processes broken down by the aforementioned disassembly means,

[0948] A generation means for generating a business flow based on the aforementioned efficiency improvement potential,

[0949] A design and development means for designing and developing an AI agent based on the aforementioned generated business flow,

[0950] Distribution means for distributing the aforementioned AI agent to the user's terminal,

[0951] A system that includes this.

[0952] (Claim 2)

[0953] The system according to claim 1, characterized in that the analysis means has a function to identify user input operations.

[0954] (Claim 3)

[0955] The system according to claim 1, wherein the means for detecting the potential for efficiency improvements in the business process has a function for detecting the frequency of repeated operations.

[0956] "Example 1"

[0957] (Claim 1)

[0958] A recording means for recording user behavior,

[0959] A processing means for compressing and encrypting the information obtained by the recording means,

[0960] A transmission means for transmitting the aforementioned information to a server,

[0961] An analysis means for analyzing the information on the server and identifying the operation content,

[0962] The analysis means breaks down the business procedure based on the results of the analysis,

[0963] A detection means for detecting the possibility of improving the efficiency of the operation based on the procedure disassembled by the aforementioned disassembly means,

[0964] A generation means for generating business procedures based on the aforementioned efficiency improvement potential,

[0965] A design and development means for designing and developing an AI automation program based on the aforementioned generated business procedures,

[0966] Distribution means for distributing the aforementioned AI automation program to the user's terminal,

[0967] A system that includes this.

[0968] (Claim 2)

[0969] The system according to claim 1, characterized in that the analysis means has a function to recognize user input.

[0970] (Claim 3)

[0971] The system according to claim 1, characterized in that the means for detecting the potential for efficiency improvements in the aforementioned work procedure has a function for detecting the frequency of repetitive operations.

[0972] "Application Example 1"

[0973] (Claim 1)

[0974] A recording means for recording user operations,

[0975] An analysis means for analyzing the information obtained by the recording means,

[0976] The analysis means includes a decomposition means that decomposes the work process based on the results of the analysis,

[0977] A detection means for detecting the potential for improving work efficiency based on the process of disassembly performed by the aforementioned disassembly means,

[0978] A generation means for generating a workflow based on the aforementioned efficiency improvement potential,

[0979] A design and development means for designing and developing an AI agent based on the generated workflow,

[0980] Distribution means for distributing the aforementioned AI agent to the user's device,

[0981] A communication means for compressing and encrypting recorded data and sending it to a server,

[0982] An execution means with control functions for automating factory operations,

[0983] A system that includes this.

[0984] (Claim 2)

[0985] The system according to claim 1, characterized in that the analysis means has a function to identify the user's input operation.

[0986] (Claim 3)

[0987] The system according to claim 1, characterized in that the means for detecting the potential for efficiency improvements in the work process has a function for detecting the frequency of repetitive work.

[0988] "Example 2 of combining an emotion engine"

[0989] (Claim 1)

[0990] A means for recording user actions and emotional states,

[0991] A transmission means for compressing and transmitting the data obtained by the acquisition means,

[0992] An analysis means for analyzing the data transmitted by the aforementioned transmission means and decomposing it into business steps and emotional states,

[0993] A detection means for detecting the efficiency of work and the optimization of emotions based on the results of the analysis by the aforementioned analysis means,

[0994] A configuration means for generating a business flow based on the detected results,

[0995] A design and generation means for designing and developing an AI agent based on the aforementioned generated business flow,

[0996] Distribution means for distributing the aforementioned AI agent to the user's device,

[0997] A system that includes this.

[0998] (Claim 2)

[0999] The system according to claim 1, characterized in that the analysis means has the function of identifying the user's input operations and emotional state.

[1000] (Claim 3)

[1001] The system according to claim 1, characterized in that the means for detecting the efficiency of the aforementioned operations and the optimization of emotions has a function for detecting the frequency of repetitive operations and fluctuations in emotions.

[1002] "Application example 2 when combining with an emotional engine"

[1003] (Claim 1)

[1004] A recording means for recording user operations,

[1005] An analysis means for analyzing the data obtained by the recording means,

[1006] The analysis means includes a decomposition means that decomposes the business process based on the results of the analysis,

[1007] A detection means for detecting areas for improving operational efficiency based on the processes broken down by the aforementioned disassembly means,

[1008] A generation means for generating a business flow based on the aforementioned efficiency improvement potential,

[1009] A design and development means for designing and developing a work support system based on the generated business flow,

[1010] Distribution means for distributing the aforementioned work support system to the user's information processing device,

[1011] A means of analyzing user emotions,

[1012] A generation means that analyzes the emotional data acquired by the aforementioned emotional analysis means and generates an emotional-based index that can be used to improve the efficiency of work,

[1013] A system that includes this.

[1014] (Claim 2)

[1015] The system according to claim 1, characterized in that the analysis means has a function to identify user input operations.

[1016] (Claim 3)

[1017] The system according to claim 1, wherein the means for detecting the potential for efficiency improvements in the business process has a function for detecting the frequency of repetitive operations and a function for adjusting work assignments based on emotional states. [Explanation of symbols]

[1018] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A recording means for recording user operations, An analysis means for analyzing the information obtained by the recording means, The analysis means includes a decomposition means that decomposes the work process based on the results of the analysis, A detection means for detecting the potential for improving work efficiency based on the process of disassembly performed by the aforementioned disassembly means, A generation means for generating a workflow based on the aforementioned efficiency improvement potential, A design and development means for designing and developing an AI agent based on the aforementioned generated workflow, Distribution means for distributing the aforementioned AI agent to the user's device, A communication means for compressing and encrypting recorded data and sending it to a server, An execution means with control functions for automating factory operations, A system that includes this.

2. The system according to claim 1, characterized in that the analysis means has a function to identify the user's input operation.

3. The system according to claim 1, characterized in that the means for detecting the potential for efficiency improvements in the work process has a function for detecting the frequency of repetitive work.

Images