system

Abstract

We provide the system. [Solution] A means for recording user actions, A means for detecting repetitive work patterns from recorded operations, A means for generating an automation script based on detected work patterns, A means of executing the generated automation script and notifying the user of the results, A means of collecting user feedback and improving automation scripts, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern system operation, it is important to promote automation. However, due to a lack of time for designing and implementing automation processes, there is a vicious cycle of ultimately relying on manual handling by humans. In such a situation, work efficiency decreases and the risk of human error increases. Therefore, there is a need for a system that automatically analyzes daily business flows and simplifies the promotion of automation.

Means for Solving the Problems

[0005] This invention provides a system that records user operations and detects repetitive work patterns based on them. The recorded operation logs are sent to a server and analyzed. Based on the analysis results, an automation script is generated, and execution efficiency is improved by further optimizing the script. The generated script is automatically executed according to a predetermined schedule, and the results are notified to the user. Furthermore, by collecting user feedback and continuously improving the script, the system's flexibility and adaptability can be enhanced. This simplifies the process to achieve automation and improves the efficiency of business operations.

[0006] A "user" refers to a person who operates a system or terminal, and is the subject from which operation history is collected.

[0007] "Operation" refers to a series of actions or command inputs performed by a user using a system or terminal, and the content of these actions is recorded.

[0008] "Means of recording" refers to functions or devices that save user actions as data and use it for subsequent analysis.

[0009] A "repetitive work pattern" is a collection of operations in which specific conditions or sequences repeatedly appear from among multiple operations, and which are detected as targets for automation.

[0010] An "automation script" refers to code or a program that executes a series of instructions or commands generated based on detected repetitive work patterns.

[0011] A "server" refers to a computer system that receives operation logs and performs analysis, script generation, and execution management.

[0012] "Execution efficiency" is a measure of how quickly and efficiently the generated automation script can perform its intended task.

[0013] "Feedback" refers to the opinions and suggestions for improvement that users provide regarding the behavior of a script and the results it produces. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention aims to automate the operation of a user-operated system. As an embodiment, it provides a system supported from three aspects: a server, a terminal, and the user.

[0036] First, the terminal monitors user actions and obtains detailed operation logs. These logs include a series of user actions, such as moving files, entering data, and launching applications. The terminal records these actions in real time and saves them as log files.

[0037] Next, the server receives operation logs sent from the terminal. The server uses algorithms, including generative AI, to analyze the logs and identify repetitive operation patterns. This analysis can detect monotonous tasks that users frequently perform.

[0038] Based on the detected patterns, the server generates an automation script. This script is converted into a pre-configured format and constructed to include the actual operating procedures and conditional branching. However, it is not provided as raw code, but rather managed as a set of procedures.

[0039] The generated scripts are managed and automatically executed by the server. This automates operations that the user previously performed manually. The server notifies the user of the results of the script execution. The notification is provided as a detailed report including the success or failure status.

[0040] For example, consider a regular data backup operation. If a user backs up files weekly from one folder to another, this operation is recorded by the terminal and analyzed on the server. The generated automation script automatically performs this backup weekly and reports the results to the user.

[0041] Furthermore, if users send feedback to the server, the script can be improved based on that information. This allows the system to continuously evolve in response to user needs.

[0042] This format allows users to automate processes without extra effort, significantly improving the efficiency of their daily work.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The terminal monitors user actions and records operation logs. Specifically, it records all user actions, including command input, file operations, and application launches. These operation logs include detailed information such as the time, type, and target of each event.

[0046] Step 2:

[0047] The terminal sends recorded operation logs to the server at regular intervals. This transmission is performed using a secure protocol, ensuring data integrity and security.

[0048] Step 3:

[0049] The server analyzes the operation log data received from the terminal. The server uses a generative AI algorithm to analyze the log data and identify repetitive operation patterns. This reveals the operations that users frequently perform that can be automated.

[0050] Step 4:

[0051] The server generates an automation script based on identified repetitive operation patterns. The generated script includes instructions for the operation steps and necessary conditions, allowing it to accurately reproduce the sequence of operations performed by the user.

[0052] Step 5:

[0053] The server executes the generated automation scripts based on the specified schedule or trigger conditions. The execution results of the scripts are recorded as logs for later reference.

[0054] Step 6:

[0055] The server notifies the user of the script's execution results. This notification is sent via email or a web interface and includes the status and details of the execution results.

[0056] Step 7:

[0057] Users can provide feedback on the script's results. This feedback is collected on the server and incorporated into the next script generation.

[0058] Step 8:

[0059] The server adjusts and improves the script based on the user feedback it collects. This allows the system to continuously evolve to better meet user needs.

[0060] (Example 1)

[0061] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0062] In modern information technology, repetitive manual operations performed by users are inefficient in terms of time and effort, hindering improvements in work efficiency. Furthermore, automating these operations requires advanced programming skills, making it difficult for the average user. Therefore, there is a need to provide a means to automatically detect and automate repetitive tasks in a user-friendly manner.

[0063] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0064] In this invention, the server includes means for using an information terminal to monitor operations and acquire data, communication means for transferring the acquired data to an analysis device, and means for the data analysis device to identify repetitive tasks and construct automated procedures using a generative AI model. This makes it possible to efficiently automate repetitive tasks that were previously performed manually by the user and to provide the results to the user through a notification function.

[0065] An "information terminal that monitors operations and acquires data" is a device that monitors a user's daily operations in real time and records the details of those operations as data.

[0066] "Communication means" refers to network technology or mechanisms for efficiently transferring data acquired by an information terminal to an analysis device.

[0067] A "data analysis device" is a computing device or system used to analyze received data and recognize the user's repetitive operation patterns.

[0068] A "generative AI model" refers to artificial intelligence technology that assists in identifying operational patterns and building automated procedures, and is generally implemented through large-scale predictive models.

[0069] An "automation procedure" refers to a set of routines built as a program based on identified operational patterns, and is a means of automatically performing everyday tasks.

[0070] "Notification functionality" refers to methods and technologies for informing users of the results of automated procedures, and is typically provided via email or as a pop-up on the device.

[0071] This invention is a system that enables the automation of repetitive tasks that users perform on a daily basis. The user's terminal uses dedicated client software to monitor user operations and acquire data in real time. This software runs on Windows and macOS® operating systems and meticulously records various operations performed by the user (such as moving files, launching applications, and entering data).

[0072] Operational data collected by the terminal is periodically transmitted to the server via communication means. The server has a data analysis device equipped with a high-performance processor and large-capacity storage, and analyzes the received data. Advanced pattern recognition technology using generative AI models is used for the analysis to identify repetitive tasks performed by the user. Custom algorithms can be implemented in this process using programming languages ​​such as Python.

[0073] Based on the analysis results, the server generates an automated procedure. This procedure is written in a specific scripting language (e.g., Python, PowerShell) and includes concrete steps to automate operations that were previously performed manually by the user. The generated procedure is executed directly from the server without burdening the user and may be operated in conjunction with project management systems and other applications.

[0074] The results of the executed procedures are fed back to the user through a notification function. This notification is delivered via email or a dedicated application's notification function and reports details of successful operations and errors.

[0075] For example, if a user moves files from one folder to another every week, this operation is recorded by the terminal and analyzed by the server. The server then generates a procedure to automate this operation on a regular weekly basis. An example of a prompt to the generated AI model might be, "Based on the regular operations the user performs every Monday, please suggest some tasks that can be automated."

[0076] These processes allow users to streamline operations and significantly reduce the time and effort required for manual operations.

[0077] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0078] Step 1:

[0079] The terminal monitors user actions in real time and records them in detail. This ensures that all user actions are saved in a data log. Input is the user's action (e.g., deleting a file, launching an application), and output is the corresponding detailed operation log file. Dedicated software running within the terminal handles this role, collecting all operation history sequentially.

[0080] Step 2:

[0081] The terminal periodically sends operation data recorded in the log to the server. The server receives the log data via a data transfer protocol and communication line. The input is the operation log collected by the terminal, and the output is the integrated log data stored on the server. This process ensures that information is reliably shared between the terminal and the server.

[0082] Step 3:

[0083] The server analyzes the received operation log data using advanced data analysis algorithms. The purpose of the analysis is to identify patterns that users frequently repeat. The input is the integrated log data stored on the server, and the output is the recognized user operation patterns. The server uses a generative AI model to learn each user's operation tendencies through the analysis of the log data.

[0084] Step 4:

[0085] The server forms a procedure for automation based on the analyzed patterns. This procedure is generated using a specific scripting language. The input is the recognized operation pattern, and the output is an operational automation script. This script is executed in a later process to automate user actions.

[0086] Step 5:

[0087] The server executes the generated automation scripts according to a schedule, thereby automating recognized operations. The input is the automation script, and the output is the result of the executed operations. The server notifies the user whether the operations were successful and whether any corrections were made, if necessary.

[0088] Step 6:

[0089] Users receive reports from the server and review the results of the automated processes. User feedback is transmitted to the server, allowing for further refinement and improvement of the automated procedures. The input is the execution result report, and the output is user feedback. This process continuously improves the system and enhances the user's work efficiency.

[0090] (Application Example 1)

[0091] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0092] Improving the operational efficiency of robotic equipment in industrial facilities requires reducing the burden of manual operation, which is labor-intensive and time-consuming, and increasing productivity through automation. However, conventional technologies require individually programming the robot's movements, which hinders this achievement. Furthermore, optimizing repetitive tasks requires advanced technology, and there is a lack of methods to perform this efficiently.

[0093] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0094] In this invention, the server includes means for recording the actions performed by a terminal device, means for detecting periodic action patterns from the recorded actions, and means for generating automated control procedures based on the detected action patterns. This makes it possible to efficiently record and analyze the actions of robotic devices in industrial facilities and automatically generate optimal action procedures.

[0095] A "terminal device" is a device equipped with input / output devices for direct operation by a user or machine, and has a function for recording its actions and operations.

[0096] An "operation pattern" refers to a characteristic pattern that is periodically repeated from a series of operations or actions performed by a terminal device or robotic device, and automated control procedures are generated based on this pattern.

[0097] An "automation control procedure" is a set of instructions or scripts generated on a server to automatically execute a series of actions of a robot or machine based on detected motion patterns.

[0098] "Feedback" refers to reactions and reports from humans or machines regarding the results of an automated control procedure, and is used to improve the automated procedure in the future.

[0099] "Work efficiency" is an indicator that measures how quickly or with minimal resources industrial facilities and robotic equipment can complete tasks during a specific period or operation, and improving this efficiency is desirable.

[0100] The system realizing this invention consists of a terminal device operated by the user and a server that analyzes the data. The terminal device has a function to record the actual operations performed by the user or machine in real time, including input operations and equipment movements. The recorded operation data is stored in the terminal device as a log file and transmitted to the server at regular intervals.

[0101] The server analyzes the operation logs sent from the terminal device. Using a generated AI model, the server identifies periodic operation patterns from this data. Based on the identified patterns, it generates automated control procedures. These procedures are designed to optimize the operation of the robotic device and are executed automatically. A feedback mechanism is also included, allowing the server to receive user feedback on the execution results and use it to improve the procedures.

[0102] For example, when robots assemble products in a factory, this inventive system meticulously records the robot's movements and automatically generates the optimal assembly procedure periodically. This ensures increased work efficiency and consistent quality.

[0103] As an example of a prompt message to the generated AI model, giving instructions such as, "Now, analyze the robot's operation logs and generate an efficient automation procedure," will set the appropriate actions.

[0104] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0105] Step 1:

[0106] The terminal device records user or machine operations in real time. It receives user operation data and machine movements as input and saves this as a detailed log file. The output is an operation log accumulated over time. This record requires accuracy and completeness because it will be used for later analysis.

[0107] Step 2:

[0108] The terminal device periodically or based on certain events transfers recorded operation logs to the server. The input is the operation log stored on the terminal device, and the output is the processed log data sent to the server. This process uses a network connection for data transmission.

[0109] Step 3:

[0110] The server analyzes log data received from terminal devices. The input is log data transmitted from the terminals, and a data analysis algorithm is used to identify periodic operating patterns. The output is the identified operating patterns and their associated information. A generative AI model is used to perform data pattern recognition, forming the basis for efficient automated control procedures.

[0111] Step 4:

[0112] The server generates automated control procedures based on the detected behavioral patterns. The input is the behavioral pattern information obtained as an analysis result, and the output is the generated automated control procedure. In this step, prompt statements are used to instruct the generating AI model. The procedure is automatically generated based on prompt statements such as, "Analyze the robot's operation log and generate an efficient automated procedure."

[0113] Step 5:

[0114] The user executes the generated automated control procedure and provides feedback on the results. The input is the control procedure received from the server, and the output is the execution result, time taken, success rate, and other feedback data. User feedback is used for further improvements, enabling optimized operation.

[0115] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0116] This invention combines an emotion engine with a system that automates user operations, enabling advanced automation that takes user emotions into account. One embodiment of this invention provides a system consisting of a server incorporating an emotion engine, a terminal for collecting operation logs, and a user utilizing these systems.

[0117] First, the terminal records the user's operations in real time. Specifically, the terminal meticulously logs all operations, including CLI commands and GUI interactions. Furthermore, the terminal uses an emotion engine to analyze the user's voice, facial expressions, and input speed to estimate their emotions. This emotion data is also recorded as part of the operation log.

[0118] Next, the server receives emotional data along with the operation logs acquired from the terminal. The received data is analyzed using a generative AI, and the recurring patterns hidden in the user's actions are cross-referenced and analyzed with the emotions at that time. Based on this analysis, the server understands what emotional states the user is most likely to experience in what situations, and uses that information to generate an automated script.

[0119] The generated automation scripts dynamically change their actions based on the user's emotional state. For example, they are designed to execute instructions that simplify tasks when the user is frustrated, and to perform the normal process when the user is calm.

[0120] This system can be adjusted to skip steps that users often find tedious, such as those involved in regular software updates. These adjustments, based on emotional data, allow users to use the system in a more comfortable environment.

[0121] Furthermore, the server collects emotionally charged feedback from users and uses it to improve the scripts. This allows the system to continuously provide more appropriate behavior that is tailored to the user's profile.

[0122] This configuration is expected to not only reduce the user's operational burden, but also significantly improve the user experience and enhance the efficiency of system operations.

[0123] The following describes the processing flow.

[0124] Step 1:

[0125] The terminal monitors user actions on the system in real time and obtains detailed operation logs. This includes keyboard input, mouse operations, and application usage history. All operations are recorded with timestamps.

[0126] Step 2:

[0127] The device analyzes the user's facial expressions and voice tone using an emotion engine to estimate the user's emotions in real time. This emotion data is recorded along with the user's actions at that time, and identifies the user's stress level, satisfaction level, and other factors.

[0128] Step 3:

[0129] The device sends recorded operation logs and emotion data to the server at regular intervals. This transmission is securely performed using an encryption protocol, ensuring data protection.

[0130] Step 4:

[0131] The server analyzes the received operation logs and sentiment data. Using generative AI, it identifies repetitive operation patterns and analyzes how they affected the user's emotions. This analysis helps to find the optimal operation flow for the user.

[0132] Step 5:

[0133] The server generates an automated script based on the analysis results. This script is designed to dynamically change in response to the user's emotional state. For example, it may include controls to skip operations that the user might find stressful.

[0134] Step 6:

[0135] The server executes the generated script. During execution, the script monitors the user's emotional state in real time and adjusts the procedure accordingly. For example, if the user is impatient, the script simplifies the task and reduces the burden on the user.

[0136] Step 7:

[0137] The server re-evaluates the script execution results and user sentiment data, and notifies the user. This notification is provided as a report that includes details about the success or failure of the operation and any changes in sentiment.

[0138] Step 8:

[0139] Users provide feedback on script execution and notification results. This feedback includes evaluations of emotional responses and opinions on the script's effectiveness.

[0140] Step 9:

[0141] The server analyzes the collected feedback and further improves the script. It optimizes the script's content and execution timing to enhance the user experience.

[0142] This entire process results in a flexible and efficient automated system that takes user emotions and behavioral patterns into account.

[0143] (Example 2)

[0144] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0145] Traditional automation systems aim to improve user work efficiency, but often fail to consider the user's emotional state. This creates a problem where there's no way to quickly improve the user experience when they feel stressed or find the work cumbersome. As a result, the user experience suffers, and the system's efficiency cannot be fully utilized.

[0146] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0147] In this invention, the server includes means for analyzing the user's emotions and adding the analysis results to the operation log, means for dynamically adjusting the automation script based on the emotion data, and means for collecting user feedback and improving the automation script. This makes it possible to provide an appropriate automation process according to the user's emotional state.

[0148] An "operation log" refers to data that records all operations performed by a user, including all interactions within the user interface.

[0149] A "repetitive work pattern" refers to a specific process or sequence of recurring tasks identified from a user's operation history.

[0150] An "automation script" refers to a set of instructions or code used to automatically execute specific operations or processes.

[0151] "Notification" refers to the act or means by which a system communicates the results or status of an operation to a user as information.

[0152] "Emotional data" refers to information that quantifies and categorizes a user's emotional state, and is generated based on factors such as voice, facial expressions, and input speed.

[0153] "Dynamic adjustment" refers to changing content or processes in real time according to the situation.

[0154] "Feedback" refers to the opinions and reactions that users provide regarding system operation and results, and the information that can be used to improve the system in the future.

[0155] This invention provides a system that automates user operations and optimizes the automation process by taking into account the user's emotional state. The system consists of a terminal, a server, and the user who utilizes it.

[0156] The terminal is specifically designed to record user actions in real time. Specifically, it meticulously records all operations on the user interface, including CLI commands and GUI interactions. It also collects data such as the user's voice, facial expressions, and input speed, which are analyzed by a built-in emotion engine to generate emotion data. This emotion data is added to the operation log and sent to the server as a comprehensive dataset.

[0157] The server is responsible for analyzing operation logs and emotion data sent from the terminal. This analysis utilizes a generative AI model, cross-referencing repetitive patterns and emotional states hidden within the operations. This identifies work patterns associated with the user's specific emotional state. Based on this identified information, the server inputs prompts into the generative AI model, generating automated scripts to enable the user to proceed with their tasks in the desired state.

[0158] The generated automation scripts have the characteristic of dynamically adjusting their execution based on the user's emotional state. For example, if the emotional data indicates that the user is irritated, the scripts are designed to include instructions that simplify the task. This can improve the user experience. Specifically, in routine system update tasks, time-consuming steps may be skipped if the user is bored.

[0159] Furthermore, the server collects emotionally charged feedback from users and uses it as data to continuously improve automated scripts. This self-improvement process ensures that the system is always tailored to the user.

[0160] An example of a prompt is, "Suggest an automated script to shorten the steps a user has to take when they are bored." This allows the generative AI model to provide specific scripts to improve the user experience.

[0161] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0162] Step 1:

[0163] The terminal records user actions in real time. First, the terminal meticulously logs all operations performed on the user interface, such as CLI commands and GUI clicks. The input is the user's actions themselves, and the output is the recorded operation log. Through this process, the terminal maintains a complete record of all user actions.

[0164] Step 2:

[0165] The device generates emotional data. The device collects the user's voice, facial expressions, and input speed, and analyzes them using an emotion engine. The input is the collected voice and facial expression data, and the output is the analyzed emotional data. This analysis process makes it possible to convert the user's emotional state into numerical values ​​and categories.

[0166] Step 3:

[0167] The terminal sends recorded operation logs and sentiment data to the server. The terminal encrypts this data and sends it to the server using the HTTP protocol. The input is operation logs and sentiment data, and the output is the data that has been securely delivered to the server. Through this transmission, all the data necessary for analysis is aggregated on the server.

[0168] Step 4:

[0169] The server analyzes the data it receives. Using a generative AI model, the server cross-references repetitive work patterns and emotional states from operation logs and emotion data. The input is the operation logs and emotion data received by the server, and the output is the relationship between identified work patterns and emotions. This analysis allows for an understanding of which emotional states are associated with specific operations.

[0170] Step 5:

[0171] The server generates an automated script. Based on the analysis results, the server inputs prompt messages into an AI model that generates an automated script tailored to the user's emotions. The input is the analyzed work pattern information, and the output is the generated automated script. For example, a prompt message such as "When the user is frustrated, suggest a script that shortens the work procedure" might be input.

[0172] Step 6:

[0173] The server executes the generated automation script and notifies the user of the results. As it runs, the script dynamically adjusts to the user's current emotional state. The input is the generated automation script, and the output is a report of the results of the performed task. For example, if the user is feeling stressed, the task steps will be simplified.

[0174] Step 7:

[0175] The user provides feedback on the system's execution results. This feedback includes information about whether the user's expectations were met and how user-friendly the system was. The input is the feedback provided by the user, and the output is a record of that feedback. This allows the system to continuously learn the user's preferences.

[0176] Step 8:

[0177] The server improves the automated scripts based on user feedback. The analyzed feedback data is used to readjust the generated AI model, modifying future scripts for better performance. The input is the feedback data, and the output is the improved automated script. This allows the system to provide users with more optimized operations over time.

[0178] (Application Example 2)

[0179] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0180] Traditional automation systems aim to reduce the user's workload, but they have limitations in improving the user experience because they do not take into account the user's emotional state. Furthermore, they lack the ability to dynamically adjust tasks based on emotions, resulting in insufficient optimization for individual users.

[0181] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0182] In this invention, the server includes means for recording user actions, means for analyzing facial expressions and voice to estimate user emotions, and means for detecting repetitive work patterns from recorded actions and emotional states. This makes it possible to improve the user experience by dynamically changing automated scripts based on the user's emotional state.

[0183] "Means for recording user actions" refers to a device or software that has the function of collecting all user actions performed on an interface as logs and storing them in a database.

[0184] "Means for detecting repetitive work patterns" refer to algorithms or programs that analyze recorded operation logs and identify patterns in which the same or similar operations are repeated.

[0185] "Means for generating automation scripts" refers to software or a system that generates instructions or code for automating tasks based on detected work patterns.

[0186] "Means of notifying information providers" refers to devices or methods that provide screen displays or audio output to directly communicate the results of the generated automation script execution to the user.

[0187] "Means for collecting user feedback and improving automation scripts" refers to a program or system that includes a process of analyzing opinions and feedback received from users and making more adaptive changes to automation scripts based on this analysis.

[0188] "Means for analyzing facial expressions and voice to estimate user emotions" refers to sensing technologies and analysis algorithms that analyze user facial expression data and voice signals to estimate their emotional state.

[0189] "Means for adjusting task content" refers to systems or programs that dynamically change automated work content according to the user's emotional state and execution environment, providing the most suitable work content.

[0190] In the system implementing this invention, the terminal first records all of the user's operations. These operations are saved in a log as CLI commands and GUI interactions. The terminal also uses software libraries such as OpenCV and DeepFace to analyze the user's facial expressions and voice data and estimate their emotions. The analysis results are recorded in a database along with the operation log and sent to the server.

[0191] Based on the received data, the server uses a generative AI model to analyze repetitive work patterns and generates automated scripts based on this information. The generated scripts are dynamically modified using Python and Node-RED, taking into account the user's emotional state. Specifically, if a user is estimated to be irritated, the system simplifies the task; conversely, if the user is judged to be focused, it performs a normal or detailed task.

[0192] Furthermore, the system notifies users of the execution results through information dissemination channels and collects feedback. This allows the automated scripts to be continuously improved based on user feedback.

[0193] For example, if a user is concerned about a specific household task (such as cleaning or cooking), they can guide the AI ​​with a prompt like this: "Generate ways to streamline household tasks while the user is feeling stressed. Infer how he needs support and recommend tasks that the robot can perform."

[0194] In this way, this invention makes it possible to provide a dynamic and efficient work environment that takes into account the user's emotional state.

[0195] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0196] Step 1:

[0197] The terminal records all operations performed by the user on the interface. Inputs include CLI commands and GUI interactions; this data is collected and stored as an operation log to prepare for subsequent processing.

[0198] Step 2:

[0199] The device analyzes the user's facial expressions and voice using software libraries such as OpenCV and DeepFace. It acquires real-time facial image and voice data from the user as input, analyzes this data to estimate the user's emotional state, and generates emotion data.

[0200] Step 3:

[0201] The terminal saves recorded operation logs and generated emotion data to a database and sends them to the server. The input consists of already generated operation logs and emotion data, which are then transferred to the server.

[0202] Step 4:

[0203] The server receives the operation logs and sentiment data and analyzes the data using a generative AI model. The input data consists of operation logs and sentiment information, which are used to detect repetitive work patterns and clarify the nature of the work.

[0204] Step 5:

[0205] The server generates automation scripts using Python scripts and Node-RED based on detected work patterns and emotional information. This creates dynamic script content that responds to the user's emotional state and outputs an appropriate execution plan.

[0206] Step 6:

[0207] The server executes the generated automation script and notifies the user of the results. The input is the generated execution plan, which is analyzed, and the results are communicated to the user in a format appropriate to the medium.

[0208] Step 7:

[0209] Users submit their thoughts and opinions to the server through the feedback form they receive. The server collects this feedback, re-enters it into the generating AI model, and uses it to improve future scripts.

[0210] This dynamically creates an efficient work environment that takes the user's emotional state into consideration.

[0211] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0212] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0213] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0214] [Second Embodiment]

[0215] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0216] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0217] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0218] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0219] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0220] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0221] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0222] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0223] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0224] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0225] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0226] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0227] This invention aims to automate the operation of a user-operated system. As an embodiment, it provides a system supported from three aspects: a server, a terminal, and the user.

[0228] First, the terminal monitors user actions and obtains detailed operation logs. These logs include a series of user actions, such as moving files, entering data, and launching applications. The terminal records these actions in real time and saves them as log files.

[0229] Next, the server receives operation logs sent from the terminal. The server uses algorithms, including generative AI, to analyze the logs and identify repetitive operation patterns. This analysis can detect monotonous tasks that users frequently perform.

[0230] Based on the detected patterns, the server generates an automation script. This script is converted into a pre-configured format and constructed to include the actual operating procedures and conditional branching. However, it is not provided as raw code, but rather managed as a set of procedures.

[0231] The generated scripts are managed and automatically executed by the server. This automates operations that the user previously performed manually. The server notifies the user of the results of the script execution. The notification is provided as a detailed report including the success or failure status.

[0232] For example, consider a regular data backup operation. If a user backs up files weekly from one folder to another, this operation is recorded by the terminal and analyzed on the server. The generated automation script automatically performs this backup weekly and reports the results to the user.

[0233] Furthermore, if users send feedback to the server, the script can be improved based on that information. This allows the system to continuously evolve in response to user needs.

[0234] This format allows users to automate processes without extra effort, significantly improving the efficiency of their daily work.

[0235] The following describes the processing flow.

[0236] Step 1:

[0237] The terminal monitors user actions and records operation logs. Specifically, it records all user actions, including command input, file operations, and application launches. These operation logs include detailed information such as the time, type, and target of each event.

[0238] Step 2:

[0239] The terminal sends recorded operation logs to the server at regular intervals. This transmission is performed using a secure protocol, ensuring data integrity and security.

[0240] Step 3:

[0241] The server analyzes the operation log data received from the terminal. The server uses a generative AI algorithm to analyze the log data and identify repetitive operation patterns. This reveals the operations that users frequently perform that can be automated.

[0242] Step 4:

[0243] The server generates an automation script based on identified repetitive operation patterns. The generated script includes instructions for the operation steps and necessary conditions, allowing it to accurately reproduce the sequence of operations performed by the user.

[0244] Step 5:

[0245] The server executes the generated automation scripts based on the specified schedule or trigger conditions. The execution results of the scripts are recorded as logs for later reference.

[0246] Step 6:

[0247] The server notifies the user of the script's execution results. This notification is sent via email or a web interface and includes the status and details of the execution results.

[0248] Step 7:

[0249] Users can provide feedback on the script's results. This feedback is collected on the server and incorporated into the next script generation.

[0250] Step 8:

[0251] The server adjusts and improves the script based on the user feedback it collects. This allows the system to continuously evolve to better meet user needs.

[0252] (Example 1)

[0253] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0254] In modern information technology, repetitive manual operations performed by users are inefficient in terms of time and effort, hindering improvements in work efficiency. Furthermore, automating these operations requires advanced programming skills, making it difficult for the average user. Therefore, there is a need to provide a means to automatically detect and automate repetitive tasks in a user-friendly manner.

[0255] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0256] In this invention, the server includes means for using an information terminal to monitor operations and acquire data, communication means for transferring the acquired data to an analysis device, and means for the data analysis device to identify repetitive tasks and construct automated procedures using a generative AI model. This makes it possible to efficiently automate repetitive tasks that were previously performed manually by the user and to provide the results to the user through a notification function.

[0257] An "information terminal that monitors operations and acquires data" is a device that monitors a user's daily operations in real time and records the details of those operations as data.

[0258] "Communication means" refers to network technology or mechanisms for efficiently transferring data acquired by an information terminal to an analysis device.

[0259] A "data analysis device" is a computing device or system used to analyze received data and recognize the user's repetitive operation patterns.

[0260] A "generative AI model" refers to artificial intelligence technology that assists in identifying operational patterns and building automated procedures, and is generally implemented through large-scale predictive models.

[0261] An "automation procedure" refers to a set of routines built as a program based on identified operational patterns, and is a means of automatically performing everyday tasks.

[0262] "Notification functionality" refers to methods and technologies for informing users of the results of automated procedures, and is typically provided via email or as a pop-up on the device.

[0263] This invention is a system that enables the automation of repetitive tasks that users perform on a daily basis. The user's terminal uses dedicated client software to monitor user operations and acquire data in real time. This software runs on Windows and macOS operating systems and meticulously records various operations performed by the user (such as moving files, launching applications, and entering data).

[0264] Operational data collected by the terminal is periodically transmitted to the server via communication means. The server has a data analysis device equipped with a high-performance processor and large-capacity storage, and analyzes the received data. Advanced pattern recognition technology using generative AI models is used for the analysis to identify repetitive tasks performed by the user. Custom algorithms can be implemented in this process using programming languages ​​such as Python.

[0265] Based on the analysis results, the server generates an automated procedure. This procedure is written in a specific scripting language (e.g., Python, PowerShell) and includes concrete steps to automate operations that were previously performed manually by the user. The generated procedure is executed directly from the server without burdening the user and may be operated in conjunction with project management systems and other applications.

[0266] The results of the executed procedures are fed back to the user through a notification function. This notification is delivered via email or a dedicated application's notification function and reports details of successful operations and errors.

[0267] For example, if a user moves files from one folder to another every week, this operation is recorded by the terminal and analyzed by the server. The server then generates a procedure to automate this operation on a regular weekly basis. An example of a prompt to the generated AI model might be, "Based on the regular operations the user performs every Monday, please suggest some tasks that can be automated."

[0268] These processes allow users to streamline operations and significantly reduce the time and effort required for manual operations.

[0269] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0270] Step 1:

[0271] The terminal monitors user actions in real time and records them in detail. This ensures that all user actions are saved in a data log. Input is the user's action (e.g., deleting a file, launching an application), and output is the corresponding detailed operation log file. Dedicated software running within the terminal handles this role, collecting all operation history sequentially.

[0272] Step 2:

[0273] The terminal periodically sends operation data recorded in the log to the server. The server receives the log data via a data transfer protocol and communication line. The input is the operation log collected by the terminal, and the output is the integrated log data stored on the server. This process ensures that information is reliably shared between the terminal and the server.

[0274] Step 3:

[0275] The server analyzes the received operation log data using advanced data analysis algorithms. The purpose of the analysis is to identify patterns that users frequently repeat. The input is the integrated log data stored on the server, and the output is the recognized user operation patterns. The server uses a generative AI model to learn each user's operation tendencies through the analysis of the log data.

[0276] Step 4:

[0277] The server forms a procedure for automation based on the analyzed patterns. This procedure is generated using a specific scripting language. The input is the recognized operation pattern, and the output is an operational automation script. This script is executed in a later process to automate user actions.

[0278] Step 5:

[0279] The server executes the generated automation script according to the schedule. As a result, the recognized operations are automatically performed. The input is the automation script, and the output is the result of the executed operation. The server notifies the user whether the operation was successful and whether any corrections were necessary as required.

[0280] Step 6:

[0281] The user receives the report from the server and checks the results of the implemented automation. Feedback from the user is transmitted to the server, and further fine-tuning and improvement of the automation procedure are carried out. The input is the execution result report, and the output is the user's feedback. As a result, the system is continuously improved, and the user's work efficiency is enhanced.

[0282] (Application Example 1)

[0283] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0284] Improving the working efficiency of robot devices in industrial facilities is required to reduce the burden of manual operations that require a lot of labor and time, and to improve productivity through automation. However, in the conventional technology, it is necessary to program the operations of robots individually, which has hindered the realization. Also, there is a problem that advanced technology is required for optimizing repetitive operations and there is a lack of methods for efficiently performing this.

[0285] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following respective means.

[0286] In this invention, the server includes means for recording the operations performed by the terminal device, means for detecting periodic operation patterns from the recorded operations, and means for generating an automated control procedure based on the detected operation patterns. As a result, it becomes possible to efficiently record and analyze the operations of the robot device in the industrial facility and automatically generate an optimal operation procedure.

[0287] The "terminal device" is a device equipped with an input / output device for direct operation by a user or a machine, and has a function for recording its operations and operations.

[0288] The "operation pattern" refers to a characteristic pattern that is periodically repeated from a series of operations and actions performed by the terminal device or the robot device, and an automated control procedure is generated based on this.

[0289] The "automated control procedure" is a procedure or script generated by the server for automatically executing a series of operations of the robot device or the machine based on the detected operation pattern.

[0290] "Feedback" is a reaction or report obtained from a human or a machine regarding the execution result of the automated control procedure, and is information used to improve the subsequent automated procedures.

[0291] "Work efficiency" is an index for measuring how quickly or with how few resources the work to be performed by the industrial facility or the robot device can be completed within a specific period or operation, and an improvement in this is required.

[0292] The system for realizing this invention is composed of a terminal device operated by a user and a server for analyzing its data. The terminal device has a function for real-time recording of the actual operations performed by the user or the machine, and this includes input operations and the operations of the device. The recorded operation data is stored in the terminal device as a log file and transmitted to the server at regular intervals.

[0293] The server analyzes the operation logs sent from the terminal device. Using a generated AI model, the server identifies periodic operation patterns from this data. Based on the identified patterns, it generates automated control procedures. These procedures are designed to optimize the operation of the robotic device and are executed automatically. A feedback mechanism is also included, allowing the server to receive user feedback on the execution results and use it to improve the procedures.

[0294] For example, when robots assemble products in a factory, this inventive system meticulously records the robot's movements and automatically generates the optimal assembly procedure periodically. This ensures increased work efficiency and consistent quality.

[0295] As an example of a prompt message to the generated AI model, giving instructions such as, "Now, analyze the robot's operation logs and generate an efficient automation procedure," will set the appropriate actions.

[0296] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0297] Step 1:

[0298] The terminal device records user or machine operations in real time. It receives user operation data and machine movements as input and saves this as a detailed log file. The output is an operation log accumulated over time. This record requires accuracy and completeness because it will be used for later analysis.

[0299] Step 2:

[0300] The terminal device periodically or based on certain events transfers recorded operation logs to the server. The input is the operation log stored on the terminal device, and the output is the processed log data sent to the server. This process uses a network connection for data transmission.

[0301] Step 3:

[0302] The server analyzes the log data received from the terminal device. The input is the log data transmitted from the terminal, and a periodic operation pattern is identified using a data analysis algorithm. The output is the identified operation pattern and its related information. Using the generated AI model, pattern recognition of the data is performed to form the basis for an efficient automated control procedure.

[0303] Step 4:

[0304] Based on the detected operation pattern, the server generates an automated control procedure. The input is the operation pattern information obtained as the analysis result, and the output is the generated automated control procedure. In this step, the generated AI model is instructed using a prompt sentence. The procedure is automatically generated based on a prompt sentence such as "Analyze the operation log of the robot and generate an efficient automated procedure."

[0305] Step 5:

[0306] The user executes the generated automated control procedure and provides feedback on the result. The input is the control procedure received from the server, and the output is feedback data such as the execution result, required time, and success rate. Feedback from the user is utilized for the next improvement, enabling optimized operation.

[0307] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion recognition model 59 and perform specific processing using the user's emotion.

[0308] The present invention combines an emotion engine with a system for automating user operations, enabling advanced automation that takes into account the user's emotion. As an embodiment thereof, a system comprising a server incorporating an emotion engine, a terminal for collecting operation logs, and a user who uses them is provided.

[0309] First, the terminal records the user's operations in real time. Specifically, the terminal meticulously logs all operations, including CLI commands and GUI interactions. Furthermore, the terminal uses an emotion engine to analyze the user's voice, facial expressions, and input speed to estimate their emotions. This emotion data is also recorded as part of the operation log.

[0310] Next, the server receives emotional data along with the operation logs acquired from the terminal. The received data is analyzed using a generative AI, and the recurring patterns hidden in the user's actions are cross-referenced and analyzed with the emotions at that time. Based on this analysis, the server understands what emotional states the user is most likely to experience in what situations, and uses that information to generate an automated script.

[0311] The generated automation scripts dynamically change their actions based on the user's emotional state. For example, they are designed to execute instructions that simplify tasks when the user is frustrated, and to perform the normal process when the user is calm.

[0312] This system can be adjusted to skip steps that users often find tedious, such as those involved in regular software updates. These adjustments, based on emotional data, allow users to use the system in a more comfortable environment.

[0313] Furthermore, the server collects emotionally charged feedback from users and uses it to improve the scripts. This allows the system to continuously provide more appropriate behavior that is tailored to the user's profile.

[0314] This configuration is expected to not only reduce the user's operational burden, but also significantly improve the user experience and enhance the efficiency of system operations.

[0315] The following describes the processing flow.

[0316] Step 1:

[0317] The terminal monitors user actions on the system in real time and obtains detailed operation logs. This includes keyboard input, mouse operations, and application usage history. All operations are recorded with timestamps.

[0318] Step 2:

[0319] The device analyzes the user's facial expressions and voice tone using an emotion engine to estimate the user's emotions in real time. This emotion data is recorded along with the user's actions at that time, and identifies the user's stress level, satisfaction level, and other factors.

[0320] Step 3:

[0321] The device sends recorded operation logs and emotion data to the server at regular intervals. This transmission is securely performed using an encryption protocol, ensuring data protection.

[0322] Step 4:

[0323] The server analyzes the received operation logs and sentiment data. Using generative AI, it identifies repetitive operation patterns and analyzes how they affected the user's emotions. This analysis helps to find the optimal operation flow for the user.

[0324] Step 5:

[0325] The server generates an automated script based on the analysis results. This script is designed to dynamically change in response to the user's emotional state. For example, it may include controls to skip operations that the user might find stressful.

[0326] Step 6:

[0327] The server executes the generated script. During execution, the script monitors the user's emotional state in real time and adjusts the procedure accordingly. For example, if the user is impatient, the script simplifies the task and reduces the burden on the user.

[0328] Step 7:

[0329] The server re-evaluates the script execution results and user sentiment data, and notifies the user. This notification is provided as a report that includes details about the success or failure of the operation and any changes in sentiment.

[0330] Step 8:

[0331] Users provide feedback on script execution and notification results. This feedback includes evaluations of emotional responses and opinions on the script's effectiveness.

[0332] Step 9:

[0333] The server analyzes the collected feedback and further improves the script. It optimizes the script's content and execution timing to enhance the user experience.

[0334] This entire process results in a flexible and efficient automated system that takes user emotions and behavioral patterns into account.

[0335] (Example 2)

[0336] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0337] Traditional automation systems aim to improve user work efficiency, but often fail to consider the user's emotional state. This creates a problem where there's no way to quickly improve the user experience when they feel stressed or find the work cumbersome. As a result, the user experience suffers, and the system's efficiency cannot be fully utilized.

[0338] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0339] In this invention, the server includes means for analyzing the user's emotions and adding the analysis results to the operation log, means for dynamically adjusting the automation script based on the emotion data, and means for collecting user feedback and improving the automation script. This makes it possible to provide an appropriate automation process according to the user's emotional state.

[0340] An "operation log" refers to data that records all operations performed by a user, including all interactions within the user interface.

[0341] A "repetitive work pattern" refers to a specific process or sequence of recurring tasks identified from a user's operation history.

[0342] An "automation script" refers to a set of instructions or code used to automatically execute specific operations or processes.

[0343] "Notification" refers to the act or means by which a system communicates the results or status of an operation to a user as information.

[0344] "Emotional data" refers to information that quantifies and categorizes a user's emotional state, and is generated based on factors such as voice, facial expressions, and input speed.

[0345] "Dynamic adjustment" refers to changing content or processes in real time according to the situation.

[0346] "Feedback" refers to the opinions and reactions that users provide regarding system operation and results, and the information that can be used to improve the system in the future.

[0347] This invention provides a system that automates user operations and optimizes the automation process by taking into account the user's emotional state. The system consists of a terminal, a server, and the user who utilizes it.

[0348] The terminal is specifically designed to record user actions in real time. Specifically, it meticulously records all operations on the user interface, including CLI commands and GUI interactions. It also collects data such as the user's voice, facial expressions, and input speed, which are analyzed by a built-in emotion engine to generate emotion data. This emotion data is added to the operation log and sent to the server as a comprehensive dataset.

[0349] The server is responsible for analyzing operation logs and emotion data sent from the terminal. This analysis utilizes a generative AI model, cross-referencing repetitive patterns and emotional states hidden within the operations. This identifies work patterns associated with the user's specific emotional state. Based on this identified information, the server inputs prompts into the generative AI model, generating automated scripts to enable the user to proceed with their tasks in the desired state.

[0350] The generated automation scripts have the characteristic of dynamically adjusting their execution based on the user's emotional state. For example, if the emotional data indicates that the user is irritated, the scripts are designed to include instructions that simplify the task. This can improve the user experience. Specifically, in routine system update tasks, time-consuming steps may be skipped if the user is bored.

[0351] Furthermore, the server collects emotionally charged feedback from users and uses it as data to continuously improve automated scripts. This self-improvement process ensures that the system is always tailored to the user.

[0352] An example of a prompt is, "Suggest an automated script to shorten the steps a user has to take when they are bored." This allows the generative AI model to provide specific scripts to improve the user experience.

[0353] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0354] Step 1:

[0355] The terminal records user actions in real time. First, the terminal meticulously logs all operations performed on the user interface, such as CLI commands and GUI clicks. The input is the user's actions themselves, and the output is the recorded operation log. Through this process, the terminal maintains a complete record of all user actions.

[0356] Step 2:

[0357] The device generates emotional data. The device collects the user's voice, facial expressions, and input speed, and analyzes them using an emotion engine. The input is the collected voice and facial expression data, and the output is the analyzed emotional data. This analysis process makes it possible to convert the user's emotional state into numerical values ​​and categories.

[0358] Step 3:

[0359] The terminal sends recorded operation logs and sentiment data to the server. The terminal encrypts this data and sends it to the server using the HTTP protocol. The input is operation logs and sentiment data, and the output is the data that has been securely delivered to the server. Through this transmission, all the data necessary for analysis is aggregated on the server.

[0360] Step 4:

[0361] The server analyzes the data it receives. Using a generative AI model, the server cross-references repetitive work patterns and emotional states from operation logs and emotion data. The input is the operation logs and emotion data received by the server, and the output is the relationship between identified work patterns and emotions. This analysis allows for an understanding of which emotional states are associated with specific operations.

[0362] Step 5:

[0363] The server generates an automated script. Based on the analysis results, the server inputs prompt messages into an AI model that generates an automated script tailored to the user's emotions. The input is the analyzed work pattern information, and the output is the generated automated script. For example, a prompt message such as "When the user is frustrated, suggest a script that shortens the work procedure" might be input.

[0364] Step 6:

[0365] The server executes the generated automation script and notifies the user of the results. As it runs, the script dynamically adjusts to the user's current emotional state. The input is the generated automation script, and the output is a report of the results of the performed task. For example, if the user is feeling stressed, the task steps will be simplified.

[0366] Step 7:

[0367] The user provides feedback on the system's execution results. This feedback includes information about whether the user's expectations were met and how user-friendly the system was. The input is the feedback provided by the user, and the output is a record of that feedback. This allows the system to continuously learn the user's preferences.

[0368] Step 8:

[0369] The server improves the automated scripts based on user feedback. The analyzed feedback data is used to readjust the generated AI model, modifying future scripts for better performance. The input is the feedback data, and the output is the improved automated script. This allows the system to provide users with more optimized operations over time.

[0370] (Application Example 2)

[0371] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0372] Traditional automation systems aim to reduce the user's workload, but they have limitations in improving the user experience because they do not take into account the user's emotional state. Furthermore, they lack the ability to dynamically adjust tasks based on emotions, resulting in insufficient optimization for individual users.

[0373] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0374] In this invention, the server includes means for recording user actions, means for analyzing facial expressions and voice to estimate user emotions, and means for detecting repetitive work patterns from recorded actions and emotional states. This makes it possible to improve the user experience by dynamically changing automated scripts based on the user's emotional state.

[0375] "Means for recording user actions" refers to a device or software that has the function of collecting all user actions performed on an interface as logs and storing them in a database.

[0376] "Means for detecting repetitive work patterns" refer to algorithms or programs that analyze recorded operation logs and identify patterns in which the same or similar operations are repeated.

[0377] "Means for generating automation scripts" refers to software or a system that generates instructions or code for automating tasks based on detected work patterns.

[0378] "Means of notifying information providers" refers to devices or methods that provide screen displays or audio output to directly communicate the results of the generated automation script execution to the user.

[0379] "Means for collecting user feedback and improving automation scripts" refers to a program or system that includes a process of analyzing opinions and feedback received from users and making more adaptive changes to automation scripts based on this analysis.

[0380] "Means for analyzing facial expressions and voice to estimate user emotions" refers to sensing technologies and analysis algorithms that analyze user facial expression data and voice signals to estimate their emotional state.

[0381] "Means for adjusting task content" refers to systems or programs that dynamically change automated work content according to the user's emotional state and execution environment, providing the most suitable work content.

[0382] In the system implementing this invention, the terminal first records all of the user's operations. These operations are saved in a log as CLI commands and GUI interactions. The terminal also uses software libraries such as OpenCV and DeepFace to analyze the user's facial expressions and voice data and estimate their emotions. The analysis results are recorded in a database along with the operation log and sent to the server.

[0383] Based on the received data, the server uses a generative AI model to analyze repetitive work patterns and generates automated scripts based on this information. The generated scripts are dynamically modified using Python and Node-RED, taking into account the user's emotional state. Specifically, if a user is estimated to be irritated, the system simplifies the task; conversely, if the user is judged to be focused, it performs a normal or detailed task.

[0384] Furthermore, the system notifies users of the execution results through information dissemination channels and collects feedback. This allows the automated scripts to be continuously improved based on user feedback.

[0385] For example, if a user is concerned about a specific household task (such as cleaning or cooking), they can guide the AI ​​with a prompt like this: "Generate ways to streamline household tasks while the user is feeling stressed. Infer how he needs support and recommend tasks that the robot can perform."

[0386] In this way, this invention makes it possible to provide a dynamic and efficient work environment that takes into account the user's emotional state.

[0387] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0388] Step 1:

[0389] The terminal records all operations performed by the user on the interface. Inputs include CLI commands and GUI interactions; this data is collected and stored as an operation log to prepare for subsequent processing.

[0390] Step 2:

[0391] The device analyzes the user's facial expressions and voice using software libraries such as OpenCV and DeepFace. It acquires real-time facial image and voice data from the user as input, analyzes this data to estimate the user's emotional state, and generates emotion data.

[0392] Step 3:

[0393] The terminal saves recorded operation logs and generated emotion data to a database and sends them to the server. The input consists of already generated operation logs and emotion data, which are then transferred to the server.

[0394] Step 4:

[0395] The server receives the operation logs and sentiment data and analyzes the data using a generative AI model. The input data consists of operation logs and sentiment information, which are used to detect repetitive work patterns and clarify the nature of the work.

[0396] Step 5:

[0397] The server generates automation scripts using Python scripts and Node-RED based on detected work patterns and emotional information. This creates dynamic script content that responds to the user's emotional state and outputs an appropriate execution plan.

[0398] Step 6:

[0399] The server executes the generated automation script and notifies the user of the results. The input is the generated execution plan, which is analyzed, and the results are communicated to the user in a format appropriate to the medium.

[0400] Step 7:

[0401] Users submit their thoughts and opinions to the server through the feedback form they receive. The server collects this feedback, re-enters it into the generating AI model, and uses it to improve future scripts.

[0402] This dynamically creates an efficient work environment that takes the user's emotional state into consideration.

[0403] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0404] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0405] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0406] [Third Embodiment]

[0407] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0408] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0409] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0410] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0411] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0412] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0413] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0414] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0415] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0416] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0417] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0418] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0419] This invention aims to automate the operation of a user-operated system. As an embodiment, it provides a system supported from three aspects: a server, a terminal, and the user.

[0420] First, the terminal monitors user actions and obtains detailed operation logs. These logs include a series of user actions, such as moving files, entering data, and launching applications. The terminal records these actions in real time and saves them as log files.

[0421] Next, the server receives operation logs sent from the terminal. The server uses algorithms, including generative AI, to analyze the logs and identify repetitive operation patterns. This analysis can detect monotonous tasks that users frequently perform.

[0422] Based on the detected patterns, the server generates an automation script. This script is converted into a pre-configured format and constructed to include the actual operating procedures and conditional branching. However, it is not provided as raw code, but rather managed as a set of procedures.

[0423] The generated scripts are managed and automatically executed by the server. This automates operations that the user previously performed manually. The server notifies the user of the results of the script execution. The notification is provided as a detailed report including the success or failure status.

[0424] For example, consider a regular data backup operation. If a user backs up files weekly from one folder to another, this operation is recorded by the terminal and analyzed on the server. The generated automation script automatically performs this backup weekly and reports the results to the user.

[0425] Furthermore, if users send feedback to the server, the script can be improved based on that information. This allows the system to continuously evolve in response to user needs.

[0426] This format allows users to automate processes without extra effort, significantly improving the efficiency of their daily work.

[0427] The following describes the processing flow.

[0428] Step 1:

[0429] The terminal monitors user actions and records operation logs. Specifically, it records all user actions, including command input, file operations, and application launches. These operation logs include detailed information such as the time, type, and target of each event.

[0430] Step 2:

[0431] The terminal sends recorded operation logs to the server at regular intervals. This transmission is performed using a secure protocol, ensuring data integrity and security.

[0432] Step 3:

[0433] The server analyzes the operation log data received from the terminal. The server uses a generative AI algorithm to analyze the log data and identify repetitive operation patterns. This reveals the operations that users frequently perform that can be automated.

[0434] Step 4:

[0435] The server generates an automation script based on identified repetitive operation patterns. The generated script includes instructions for the operation steps and necessary conditions, allowing it to accurately reproduce the sequence of operations performed by the user.

[0436] Step 5:

[0437] The server executes the generated automation scripts based on the specified schedule or trigger conditions. The execution results of the scripts are recorded as logs for later reference.

[0438] Step 6:

[0439] The server notifies the user of the script's execution results. This notification is sent via email or a web interface and includes the status and details of the execution results.

[0440] Step 7:

[0441] Users can provide feedback on the script's results. This feedback is collected on the server and incorporated into the next script generation.

[0442] Step 8:

[0443] The server adjusts and improves the script based on the user feedback it collects. This allows the system to continuously evolve to better meet user needs.

[0444] (Example 1)

[0445] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0446] In modern information technology, repetitive manual operations performed by users are inefficient in terms of time and effort, hindering improvements in work efficiency. Furthermore, automating these operations requires advanced programming skills, making it difficult for the average user. Therefore, there is a need to provide a means to automatically detect and automate repetitive tasks in a user-friendly manner.

[0447] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0448] In this invention, the server includes means for using an information terminal to monitor operations and acquire data, communication means for transferring the acquired data to an analysis device, and means for the data analysis device to identify repetitive tasks and construct automated procedures using a generative AI model. This makes it possible to efficiently automate repetitive tasks that were previously performed manually by the user and to provide the results to the user through a notification function.

[0449] An "information terminal that monitors operations and acquires data" is a device that monitors a user's daily operations in real time and records the details of those operations as data.

[0450] "Communication means" refers to network technology or mechanisms for efficiently transferring data acquired by an information terminal to an analysis device.

[0451] A "data analysis device" is a computing device or system used to analyze received data and recognize the user's repetitive operation patterns.

[0452] A "generative AI model" refers to artificial intelligence technology that assists in identifying operational patterns and building automated procedures, and is generally implemented through large-scale predictive models.

[0453] An "automation procedure" refers to a set of routines built as a program based on identified operational patterns, and is a means of automatically performing everyday tasks.

[0454] "Notification functionality" refers to methods and technologies for informing users of the results of automated procedures, and is typically provided via email or as a pop-up on the device.

[0455] This invention is a system that enables the automation of repetitive tasks that users perform on a daily basis. The user's terminal uses dedicated client software to monitor user operations and acquire data in real time. This software runs on Windows and macOS operating systems and meticulously records various operations performed by the user (such as moving files, launching applications, and entering data).

[0456] Operational data collected by the terminal is periodically transmitted to the server via communication means. The server has a data analysis device equipped with a high-performance processor and large-capacity storage, and analyzes the received data. Advanced pattern recognition technology using generative AI models is used for the analysis to identify repetitive tasks performed by the user. Custom algorithms can be implemented in this process using programming languages ​​such as Python.

[0457] Based on the analysis results, the server generates an automated procedure. This procedure is written in a specific scripting language (e.g., Python, PowerShell) and includes concrete steps to automate operations that were previously performed manually by the user. The generated procedure is executed directly from the server without burdening the user and may be operated in conjunction with project management systems and other applications.

[0458] The results of the executed procedures are fed back to the user through a notification function. This notification is delivered via email or a dedicated application's notification function and reports details of successful operations and errors.

[0459] For example, if a user moves files from one folder to another every week, this operation is recorded by the terminal and analyzed by the server. The server then generates a procedure to automate this operation on a regular weekly basis. An example of a prompt to the generated AI model might be, "Based on the regular operations the user performs every Monday, please suggest some tasks that can be automated."

[0460] These processes allow users to streamline operations and significantly reduce the time and effort required for manual operations.

[0461] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0462] Step 1:

[0463] The terminal monitors user actions in real time and records them in detail. This ensures that all user actions are saved in a data log. Input is the user's action (e.g., deleting a file, launching an application), and output is the corresponding detailed operation log file. Dedicated software running within the terminal handles this role, collecting all operation history sequentially.

[0464] Step 2:

[0465] The terminal periodically sends operation data recorded in the log to the server. The server receives the log data via a data transfer protocol and communication line. The input is the operation log collected by the terminal, and the output is the integrated log data stored on the server. This process ensures that information is reliably shared between the terminal and the server.

[0466] Step 3:

[0467] The server analyzes the received operation log data using advanced data analysis algorithms. The purpose of the analysis is to identify patterns that users frequently repeat. The input is the integrated log data stored on the server, and the output is the recognized user operation patterns. The server uses a generative AI model to learn each user's operation tendencies through the analysis of the log data.

[0468] Step 4:

[0469] The server forms a procedure for automation based on the analyzed patterns. This procedure is generated using a specific scripting language. The input is the recognized operation pattern, and the output is an operational automation script. This script is executed in a later process to automate user actions.

[0470] Step 5:

[0471] The server executes the generated automation scripts according to a schedule, thereby automating recognized operations. The input is the automation script, and the output is the result of the executed operations. The server notifies the user whether the operations were successful and whether any corrections were made, if necessary.

[0472] Step 6:

[0473] Users receive reports from the server and review the results of the automated processes. User feedback is transmitted to the server, allowing for further refinement and improvement of the automated procedures. The input is the execution result report, and the output is user feedback. This process continuously improves the system and enhances the user's work efficiency.

[0474] (Application Example 1)

[0475] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0476] Improving the operational efficiency of robotic equipment in industrial facilities requires reducing the burden of manual operation, which is labor-intensive and time-consuming, and increasing productivity through automation. However, conventional technologies require individually programming the robot's movements, which hinders this achievement. Furthermore, optimizing repetitive tasks requires advanced technology, and there is a lack of methods to perform this efficiently.

[0477] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0478] In this invention, the server includes means for recording the actions performed by a terminal device, means for detecting periodic action patterns from the recorded actions, and means for generating automated control procedures based on the detected action patterns. This makes it possible to efficiently record and analyze the actions of robotic devices in industrial facilities and automatically generate optimal action procedures.

[0479] A "terminal device" is a device equipped with input / output devices for direct operation by a user or machine, and has a function for recording its actions and operations.

[0480] An "operation pattern" refers to a characteristic pattern that is periodically repeated from a series of operations or actions performed by a terminal device or robotic device, and automated control procedures are generated based on this pattern.

[0481] An "automation control procedure" is a set of instructions or scripts generated on a server to automatically execute a series of actions of a robot or machine based on detected motion patterns.

[0482] "Feedback" refers to reactions and reports from humans or machines regarding the results of an automated control procedure, and is used to improve the automated procedure in the future.

[0483] "Work efficiency" is an indicator that measures how quickly or with minimal resources industrial facilities and robotic equipment can complete tasks during a specific period or operation, and improving this efficiency is desirable.

[0484] The system realizing this invention consists of a terminal device operated by the user and a server that analyzes the data. The terminal device has a function to record the actual operations performed by the user or machine in real time, including input operations and equipment movements. The recorded operation data is stored in the terminal device as a log file and transmitted to the server at regular intervals.

[0485] The server analyzes the operation logs sent from the terminal device. Using a generated AI model, the server identifies periodic operation patterns from this data. Based on the identified patterns, it generates automated control procedures. These procedures are designed to optimize the operation of the robotic device and are executed automatically. A feedback mechanism is also included, allowing the server to receive user feedback on the execution results and use it to improve the procedures.

[0486] For example, when robots assemble products in a factory, this inventive system meticulously records the robot's movements and automatically generates the optimal assembly procedure periodically. This ensures increased work efficiency and consistent quality.

[0487] As an example of a prompt message to the generated AI model, giving instructions such as, "Now, analyze the robot's operation logs and generate an efficient automation procedure," will set the appropriate actions.

[0488] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0489] Step 1:

[0490] The terminal device records user or machine operations in real time. It receives user operation data and machine movements as input and saves this as a detailed log file. The output is an operation log accumulated over time. This record requires accuracy and completeness because it will be used for later analysis.

[0491] Step 2:

[0492] The terminal device periodically or based on certain events transfers recorded operation logs to the server. The input is the operation log stored on the terminal device, and the output is the processed log data sent to the server. This process uses a network connection for data transmission.

[0493] Step 3:

[0494] The server analyzes log data received from terminal devices. The input is log data transmitted from the terminals, and a data analysis algorithm is used to identify periodic operating patterns. The output is the identified operating patterns and their associated information. A generative AI model is used to perform data pattern recognition, forming the basis for efficient automated control procedures.

[0495] Step 4:

[0496] The server generates automated control procedures based on the detected behavioral patterns. The input is the behavioral pattern information obtained as an analysis result, and the output is the generated automated control procedure. In this step, prompt statements are used to instruct the generating AI model. The procedure is automatically generated based on prompt statements such as, "Analyze the robot's operation log and generate an efficient automated procedure."

[0497] Step 5:

[0498] The user executes the generated automated control procedure and provides feedback on the results. The input is the control procedure received from the server, and the output is the execution result, time taken, success rate, and other feedback data. User feedback is used for further improvements, enabling optimized operation.

[0499] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0500] This invention combines an emotion engine with a system that automates user operations, enabling advanced automation that takes user emotions into account. One embodiment of this invention provides a system consisting of a server incorporating an emotion engine, a terminal for collecting operation logs, and a user utilizing these systems.

[0501] First, the terminal records the user's operations in real time. Specifically, the terminal meticulously logs all operations, including CLI commands and GUI interactions. Furthermore, the terminal uses an emotion engine to analyze the user's voice, facial expressions, and input speed to estimate their emotions. This emotion data is also recorded as part of the operation log.

[0502] Next, the server receives emotional data along with the operation logs acquired from the terminal. The received data is analyzed using a generative AI, and the recurring patterns hidden in the user's actions are cross-referenced and analyzed with the emotions at that time. Based on this analysis, the server understands what emotional states the user is most likely to experience in what situations, and uses that information to generate an automated script.

[0503] The generated automation scripts dynamically change their actions based on the user's emotional state. For example, they are designed to execute instructions that simplify tasks when the user is frustrated, and to perform the normal process when the user is calm.

[0504] This system can be adjusted to skip steps that users often find tedious, such as those involved in regular software updates. These adjustments, based on emotional data, allow users to use the system in a more comfortable environment.

[0505] Furthermore, the server collects emotionally charged feedback from users and uses it to improve the scripts. This allows the system to continuously provide more appropriate behavior that is tailored to the user's profile.

[0506] This configuration is expected to not only reduce the user's operational burden, but also significantly improve the user experience and enhance the efficiency of system operations.

[0507] The following describes the processing flow.

[0508] Step 1:

[0509] The terminal monitors user actions on the system in real time and obtains detailed operation logs. This includes keyboard input, mouse operations, and application usage history. All operations are recorded with timestamps.

[0510] Step 2:

[0511] The device analyzes the user's facial expressions and voice tone using an emotion engine to estimate the user's emotions in real time. This emotion data is recorded along with the user's actions at that time, and identifies the user's stress level, satisfaction level, and other factors.

[0512] Step 3:

[0513] The device sends recorded operation logs and emotion data to the server at regular intervals. This transmission is securely performed using an encryption protocol, ensuring data protection.

[0514] Step 4:

[0515] The server analyzes the received operation logs and sentiment data. Using generative AI, it identifies repetitive operation patterns and analyzes how they affected the user's emotions. This analysis helps to find the optimal operation flow for the user.

[0516] Step 5:

[0517] The server generates an automated script based on the analysis results. This script is designed to dynamically change in response to the user's emotional state. For example, it may include controls to skip operations that the user might find stressful.

[0518] Step 6:

[0519] The server executes the generated script. During execution, the script monitors the user's emotional state in real time and adjusts the procedure accordingly. For example, if the user is impatient, the script simplifies the task and reduces the burden on the user.

[0520] Step 7:

[0521] The server re-evaluates the script execution results and user sentiment data, and notifies the user. This notification is provided as a report that includes details about the success or failure of the operation and any changes in sentiment.

[0522] Step 8:

[0523] Users provide feedback on script execution and notification results. This feedback includes evaluations of emotional responses and opinions on the script's effectiveness.

[0524] Step 9:

[0525] The server analyzes the collected feedback and further improves the script. It optimizes the script's content and execution timing to enhance the user experience.

[0526] This entire process results in a flexible and efficient automated system that takes user emotions and behavioral patterns into account.

[0527] (Example 2)

[0528] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0529] Traditional automation systems aim to improve user work efficiency, but often fail to consider the user's emotional state. This creates a problem where there's no way to quickly improve the user experience when they feel stressed or find the work cumbersome. As a result, the user experience suffers, and the system's efficiency cannot be fully utilized.

[0530] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0531] In this invention, the server includes means for analyzing the user's emotions and adding the analysis results to the operation log, means for dynamically adjusting the automation script based on the emotion data, and means for collecting user feedback and improving the automation script. This makes it possible to provide an appropriate automation process according to the user's emotional state.

[0532] An "operation log" refers to data that records all operations performed by a user, including all interactions within the user interface.

[0533] A "repetitive work pattern" refers to a specific process or sequence of recurring tasks identified from a user's operation history.

[0534] An "automation script" refers to a set of instructions or code used to automatically execute specific operations or processes.

[0535] "Notification" refers to the act or means by which a system communicates the results or status of an operation to a user as information.

[0536] "Emotional data" refers to information that quantifies and categorizes a user's emotional state, and is generated based on factors such as voice, facial expressions, and input speed.

[0537] "Dynamic adjustment" refers to changing content or processes in real time according to the situation.

[0538] "Feedback" refers to the opinions and reactions that users provide regarding system operation and results, and the information that can be used to improve the system in the future.

[0539] This invention provides a system that automates user operations and optimizes the automation process by taking into account the user's emotional state. The system consists of a terminal, a server, and the user who utilizes it.

[0540] The terminal is specifically designed to record user actions in real time. Specifically, it meticulously records all operations on the user interface, including CLI commands and GUI interactions. It also collects data such as the user's voice, facial expressions, and input speed, which are analyzed by a built-in emotion engine to generate emotion data. This emotion data is added to the operation log and sent to the server as a comprehensive dataset.

[0541] The server is responsible for analyzing operation logs and emotion data sent from the terminal. This analysis utilizes a generative AI model, cross-referencing repetitive patterns and emotional states hidden within the operations. This identifies work patterns associated with the user's specific emotional state. Based on this identified information, the server inputs prompts into the generative AI model, generating automated scripts to enable the user to proceed with their tasks in the desired state.

[0542] The generated automation scripts have the characteristic of dynamically adjusting their execution based on the user's emotional state. For example, if the emotional data indicates that the user is irritated, the scripts are designed to include instructions that simplify the task. This can improve the user experience. Specifically, in routine system update tasks, time-consuming steps may be skipped if the user is bored.

[0543] Furthermore, the server collects emotionally charged feedback from users and uses it as data to continuously improve automated scripts. This self-improvement process ensures that the system is always tailored to the user.

[0544] An example of a prompt is, "Suggest an automated script to shorten the steps a user has to take when they are bored." This allows the generative AI model to provide specific scripts to improve the user experience.

[0545] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0546] Step 1:

[0547] The terminal records user actions in real time. First, the terminal meticulously logs all operations performed on the user interface, such as CLI commands and GUI clicks. The input is the user's actions themselves, and the output is the recorded operation log. Through this process, the terminal maintains a complete record of all user actions.

[0548] Step 2:

[0549] The device generates emotional data. The device collects the user's voice, facial expressions, and input speed, and analyzes them using an emotion engine. The input is the collected voice and facial expression data, and the output is the analyzed emotional data. This analysis process makes it possible to convert the user's emotional state into numerical values ​​and categories.

[0550] Step 3:

[0551] The terminal sends recorded operation logs and sentiment data to the server. The terminal encrypts this data and sends it to the server using the HTTP protocol. The input is operation logs and sentiment data, and the output is the data that has been securely delivered to the server. Through this transmission, all the data necessary for analysis is aggregated on the server.

[0552] Step 4:

[0553] The server analyzes the data it receives. Using a generative AI model, the server cross-references repetitive work patterns and emotional states from operation logs and emotion data. The input is the operation logs and emotion data received by the server, and the output is the relationship between identified work patterns and emotions. This analysis allows for an understanding of which emotional states are associated with specific operations.

[0554] Step 5:

[0555] The server generates an automated script. Based on the analysis results, the server inputs prompt messages into an AI model that generates an automated script tailored to the user's emotions. The input is the analyzed work pattern information, and the output is the generated automated script. For example, a prompt message such as "When the user is frustrated, suggest a script that shortens the work procedure" might be input.

[0556] Step 6:

[0557] The server executes the generated automation script and notifies the user of the results. As it runs, the script dynamically adjusts to the user's current emotional state. The input is the generated automation script, and the output is a report of the results of the performed task. For example, if the user is feeling stressed, the task steps will be simplified.

[0558] Step 7:

[0559] The user provides feedback on the system's execution results. This feedback includes information about whether the user's expectations were met and how user-friendly the system was. The input is the feedback provided by the user, and the output is a record of that feedback. This allows the system to continuously learn the user's preferences.

[0560] Step 8:

[0561] The server improves the automated scripts based on user feedback. The analyzed feedback data is used to readjust the generated AI model, modifying future scripts for better performance. The input is the feedback data, and the output is the improved automated script. This allows the system to provide users with more optimized operations over time.

[0562] (Application Example 2)

[0563] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0564] Traditional automation systems aim to reduce the user's workload, but they have limitations in improving the user experience because they do not take into account the user's emotional state. Furthermore, they lack the ability to dynamically adjust tasks based on emotions, resulting in insufficient optimization for individual users.

[0565] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0566] In this invention, the server includes means for recording user actions, means for analyzing facial expressions and voice to estimate user emotions, and means for detecting repetitive work patterns from recorded actions and emotional states. This makes it possible to improve the user experience by dynamically changing automated scripts based on the user's emotional state.

[0567] "Means for recording user actions" refers to a device or software that has the function of collecting all user actions performed on an interface as logs and storing them in a database.

[0568] "Means for detecting repetitive work patterns" refer to algorithms or programs that analyze recorded operation logs and identify patterns in which the same or similar operations are repeated.

[0569] "Means for generating automation scripts" refers to software or a system that generates instructions or code for automating tasks based on detected work patterns.

[0570] "Means of notifying information providers" refers to devices or methods that provide screen displays or audio output to directly communicate the results of the generated automation script execution to the user.

[0571] "Means for collecting user feedback and improving automation scripts" refers to a program or system that includes a process of analyzing opinions and feedback received from users and making more adaptive changes to automation scripts based on this analysis.

[0572] "Means for analyzing facial expressions and voice to estimate user emotions" refers to sensing technologies and analysis algorithms that analyze user facial expression data and voice signals to estimate their emotional state.

[0573] "Means for adjusting task content" refers to systems or programs that dynamically change automated work content according to the user's emotional state and execution environment, providing the most suitable work content.

[0574] In the system implementing this invention, the terminal first records all of the user's operations. These operations are saved in a log as CLI commands and GUI interactions. The terminal also uses software libraries such as OpenCV and DeepFace to analyze the user's facial expressions and voice data and estimate their emotions. The analysis results are recorded in a database along with the operation log and sent to the server.

[0575] Based on the received data, the server uses a generative AI model to analyze repetitive work patterns and generates automated scripts based on this information. The generated scripts are dynamically modified using Python and Node-RED, taking into account the user's emotional state. Specifically, if a user is estimated to be irritated, the system simplifies the task; conversely, if the user is judged to be focused, it performs a normal or detailed task.

[0576] Furthermore, the system notifies users of the execution results through information dissemination channels and collects feedback. This allows the automated scripts to be continuously improved based on user feedback.

[0577] For example, if a user is concerned about a specific household task (such as cleaning or cooking), they can guide the AI ​​with a prompt like this: "Generate ways to streamline household tasks while the user is feeling stressed. Infer how he needs support and recommend tasks that the robot can perform."

[0578] In this way, this invention makes it possible to provide a dynamic and efficient work environment that takes into account the user's emotional state.

[0579] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0580] Step 1:

[0581] The terminal records all operations performed by the user on the interface. Inputs include CLI commands and GUI interactions; this data is collected and stored as an operation log to prepare for subsequent processing.

[0582] Step 2:

[0583] The device analyzes the user's facial expressions and voice using software libraries such as OpenCV and DeepFace. It acquires real-time facial image and voice data from the user as input, analyzes this data to estimate the user's emotional state, and generates emotion data.

[0584] Step 3:

[0585] The terminal saves recorded operation logs and generated emotion data to a database and sends them to the server. The input consists of already generated operation logs and emotion data, which are then transferred to the server.

[0586] Step 4:

[0587] The server receives the operation logs and sentiment data and analyzes the data using a generative AI model. The input data consists of operation logs and sentiment information, which are used to detect repetitive work patterns and clarify the nature of the work.

[0588] Step 5:

[0589] The server generates automation scripts using Python scripts and Node-RED based on detected work patterns and emotional information. This creates dynamic script content that responds to the user's emotional state and outputs an appropriate execution plan.

[0590] Step 6:

[0591] The server executes the generated automation script and notifies the user of the results. The input is the generated execution plan, which is analyzed, and the results are communicated to the user in a format appropriate to the medium.

[0592] Step 7:

[0593] Users submit their thoughts and opinions to the server through the feedback form they receive. The server collects this feedback, re-enters it into the generating AI model, and uses it to improve future scripts.

[0594] This dynamically creates an efficient work environment that takes the user's emotional state into consideration.

[0595] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0596] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0597] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0598] [Fourth Embodiment]

[0599] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0600] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0601] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0602] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0603] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0604] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0605] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0606] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0607] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0608] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0609] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0610] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0611] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0612] This invention aims to automate the operation of a user-operated system. As an embodiment, it provides a system supported from three aspects: a server, a terminal, and the user.

[0613] First, the terminal monitors user actions and obtains detailed operation logs. These logs include a series of user actions, such as moving files, entering data, and launching applications. The terminal records these actions in real time and saves them as log files.

[0614] Next, the server receives operation logs sent from the terminal. The server uses algorithms, including generative AI, to analyze the logs and identify repetitive operation patterns. This analysis can detect monotonous tasks that users frequently perform.

[0615] Based on the detected patterns, the server generates an automation script. This script is converted into a pre-configured format and constructed to include the actual operating procedures and conditional branching. However, it is not provided as raw code, but rather managed as a set of procedures.

[0616] The generated scripts are managed and automatically executed by the server. This automates operations that the user previously performed manually. The server notifies the user of the results of the script execution. The notification is provided as a detailed report including the success or failure status.

[0617] For example, consider a regular data backup operation. If a user backs up files weekly from one folder to another, this operation is recorded by the terminal and analyzed on the server. The generated automation script automatically performs this backup weekly and reports the results to the user.

[0618] Furthermore, if users send feedback to the server, the script can be improved based on that information. This allows the system to continuously evolve in response to user needs.

[0619] This format allows users to automate processes without extra effort, significantly improving the efficiency of their daily work.

[0620] The following describes the processing flow.

[0621] Step 1:

[0622] The terminal monitors user actions and records operation logs. Specifically, it records all user actions, including command input, file operations, and application launches. These operation logs include detailed information such as the time, type, and target of each event.

[0623] Step 2:

[0624] The terminal sends recorded operation logs to the server at regular intervals. This transmission is performed using a secure protocol, ensuring data integrity and security.

[0625] Step 3:

[0626] The server analyzes the operation log data received from the terminal. The server uses a generative AI algorithm to analyze the log data and identify repetitive operation patterns. This reveals the operations that users frequently perform that can be automated.

[0627] Step 4:

[0628] The server generates an automation script based on identified repetitive operation patterns. The generated script includes instructions for the operation steps and necessary conditions, allowing it to accurately reproduce the sequence of operations performed by the user.

[0629] Step 5:

[0630] The server executes the generated automation scripts based on the specified schedule or trigger conditions. The execution results of the scripts are recorded as logs for later reference.

[0631] Step 6:

[0632] The server notifies the user of the script's execution results. This notification is sent via email or a web interface and includes the status and details of the execution results.

[0633] Step 7:

[0634] Users can provide feedback on the script's results. This feedback is collected on the server and incorporated into the next script generation.

[0635] Step 8:

[0636] The server adjusts and improves the script based on the user feedback it collects. This allows the system to continuously evolve to better meet user needs.

[0637] (Example 1)

[0638] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0639] In modern information technology, repetitive manual operations performed by users are inefficient in terms of time and effort, hindering improvements in work efficiency. Furthermore, automating these operations requires advanced programming skills, making it difficult for the average user. Therefore, there is a need to provide a means to automatically detect and automate repetitive tasks in a user-friendly manner.

[0640] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0641] In this invention, the server includes means for using an information terminal to monitor operations and acquire data, communication means for transferring the acquired data to an analysis device, and means for the data analysis device to identify repetitive tasks and construct automated procedures using a generative AI model. This makes it possible to efficiently automate repetitive tasks that were previously performed manually by the user and to provide the results to the user through a notification function.

[0642] An "information terminal that monitors operations and acquires data" is a device that monitors a user's daily operations in real time and records the details of those operations as data.

[0643] "Communication means" refers to network technology or mechanisms for efficiently transferring data acquired by an information terminal to an analysis device.

[0644] A "data analysis device" is a computing device or system used to analyze received data and recognize the user's repetitive operation patterns.

[0645] A "generative AI model" refers to artificial intelligence technology that assists in identifying operational patterns and building automated procedures, and is generally implemented through large-scale predictive models.

[0646] An "automation procedure" refers to a set of routines built as a program based on identified operational patterns, and is a means of automatically performing everyday tasks.

[0647] "Notification functionality" refers to methods and technologies for informing users of the results of automated procedures, and is typically provided via email or as a pop-up on the device.

[0648] This invention is a system that enables the automation of repetitive tasks that users perform on a daily basis. The user's terminal uses dedicated client software to monitor user operations and acquire data in real time. This software runs on Windows and macOS operating systems and meticulously records various operations performed by the user (such as moving files, launching applications, and entering data).

[0649] Operational data collected by the terminal is periodically transmitted to the server via communication means. The server has a data analysis device equipped with a high-performance processor and large-capacity storage, and analyzes the received data. Advanced pattern recognition technology using generative AI models is used for the analysis to identify repetitive tasks performed by the user. Custom algorithms can be implemented in this process using programming languages ​​such as Python.

[0650] Based on the analysis results, the server generates an automated procedure. This procedure is written in a specific scripting language (e.g., Python, PowerShell) and includes concrete steps to automate operations that were previously performed manually by the user. The generated procedure is executed directly from the server without burdening the user and may be operated in conjunction with project management systems and other applications.

[0651] The results of the executed procedures are fed back to the user through a notification function. This notification is delivered via email or a dedicated application's notification function and reports details of successful operations and errors.

[0652] For example, if a user moves files from one folder to another every week, this operation is recorded by the terminal and analyzed by the server. The server then generates a procedure to automate this operation on a regular weekly basis. An example of a prompt to the generated AI model might be, "Based on the regular operations the user performs every Monday, please suggest some tasks that can be automated."

[0653] These processes allow users to streamline operations and significantly reduce the time and effort required for manual operations.

[0654] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0655] Step 1:

[0656] The terminal monitors user actions in real time and records them in detail. This ensures that all user actions are saved in a data log. Input is the user's action (e.g., deleting a file, launching an application), and output is the corresponding detailed operation log file. Dedicated software running within the terminal handles this role, collecting all operation history sequentially.

[0657] Step 2:

[0658] The terminal periodically sends operation data recorded in the log to the server. The server receives the log data via a data transfer protocol and communication line. The input is the operation log collected by the terminal, and the output is the integrated log data stored on the server. This process ensures that information is reliably shared between the terminal and the server.

[0659] Step 3:

[0660] The server analyzes the received operation log data using advanced data analysis algorithms. The purpose of the analysis is to identify patterns that users frequently repeat. The input is the integrated log data stored on the server, and the output is the recognized user operation patterns. The server uses a generative AI model to learn each user's operation tendencies through the analysis of the log data.

[0661] Step 4:

[0662] The server forms a procedure for automation based on the analyzed patterns. This procedure is generated using a specific scripting language. The input is the recognized operation pattern, and the output is an operational automation script. This script is executed in a later process to automate user actions.

[0663] Step 5:

[0664] The server executes the generated automation scripts according to a schedule, thereby automating recognized operations. The input is the automation script, and the output is the result of the executed operations. The server notifies the user whether the operations were successful and whether any corrections were made, if necessary.

[0665] Step 6:

[0666] Users receive reports from the server and review the results of the automated processes. User feedback is transmitted to the server, allowing for further refinement and improvement of the automated procedures. The input is the execution result report, and the output is user feedback. This process continuously improves the system and enhances the user's work efficiency.

[0667] (Application Example 1)

[0668] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0669] Improving the operational efficiency of robotic equipment in industrial facilities requires reducing the burden of manual operation, which is labor-intensive and time-consuming, and increasing productivity through automation. However, conventional technologies require individually programming the robot's movements, which hinders this achievement. Furthermore, optimizing repetitive tasks requires advanced technology, and there is a lack of methods to perform this efficiently.

[0670] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0671] In this invention, the server includes means for recording the actions performed by a terminal device, means for detecting periodic action patterns from the recorded actions, and means for generating automated control procedures based on the detected action patterns. This makes it possible to efficiently record and analyze the actions of robotic devices in industrial facilities and automatically generate optimal action procedures.

[0672] A "terminal device" is a device equipped with input / output devices for direct operation by a user or machine, and has a function for recording its actions and operations.

[0673] An "operation pattern" refers to a characteristic pattern that is periodically repeated from a series of operations or actions performed by a terminal device or robotic device, and automated control procedures are generated based on this pattern.

[0674] An "automation control procedure" is a set of instructions or scripts generated on a server to automatically execute a series of actions of a robot or machine based on detected motion patterns.

[0675] "Feedback" refers to reactions and reports from humans or machines regarding the results of an automated control procedure, and is used to improve the automated procedure in the future.

[0676] "Work efficiency" is an indicator that measures how quickly or with minimal resources industrial facilities and robotic equipment can complete tasks during a specific period or operation, and improving this efficiency is desirable.

[0677] The system realizing this invention consists of a terminal device operated by the user and a server that analyzes the data. The terminal device has a function to record the actual operations performed by the user or machine in real time, including input operations and equipment movements. The recorded operation data is stored in the terminal device as a log file and transmitted to the server at regular intervals.

[0678] The server analyzes the operation logs sent from the terminal device. Using a generated AI model, the server identifies periodic operation patterns from this data. Based on the identified patterns, it generates automated control procedures. These procedures are designed to optimize the operation of the robotic device and are executed automatically. A feedback mechanism is also included, allowing the server to receive user feedback on the execution results and use it to improve the procedures.

[0679] For example, when robots assemble products in a factory, this inventive system meticulously records the robot's movements and automatically generates the optimal assembly procedure periodically. This ensures increased work efficiency and consistent quality.

[0680] As an example of a prompt message to the generated AI model, giving instructions such as, "Now, analyze the robot's operation logs and generate an efficient automation procedure," will set the appropriate actions.

[0681] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0682] Step 1:

[0683] The terminal device records user or machine operations in real time. It receives user operation data and machine movements as input and saves this as a detailed log file. The output is an operation log accumulated over time. This record requires accuracy and completeness because it will be used for later analysis.

[0684] Step 2:

[0685] The terminal device periodically or based on certain events transfers recorded operation logs to the server. The input is the operation log stored on the terminal device, and the output is the processed log data sent to the server. This process uses a network connection for data transmission.

[0686] Step 3:

[0687] The server analyzes log data received from terminal devices. The input is log data transmitted from the terminals, and a data analysis algorithm is used to identify periodic operating patterns. The output is the identified operating patterns and their associated information. A generative AI model is used to perform data pattern recognition, forming the basis for efficient automated control procedures.

[0688] Step 4:

[0689] The server generates automated control procedures based on the detected behavioral patterns. The input is the behavioral pattern information obtained as an analysis result, and the output is the generated automated control procedure. In this step, prompt statements are used to instruct the generating AI model. The procedure is automatically generated based on prompt statements such as, "Analyze the robot's operation log and generate an efficient automated procedure."

[0690] Step 5:

[0691] The user executes the generated automated control procedure and provides feedback on the results. The input is the control procedure received from the server, and the output is the execution result, time taken, success rate, and other feedback data. User feedback is used for further improvements, enabling optimized operation.

[0692] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0693] This invention combines an emotion engine with a system that automates user operations, enabling advanced automation that takes user emotions into account. One embodiment of this invention provides a system consisting of a server incorporating an emotion engine, a terminal for collecting operation logs, and a user utilizing these systems.

[0694] First, the terminal records the user's operations in real time. Specifically, the terminal meticulously logs all operations, including CLI commands and GUI interactions. Furthermore, the terminal uses an emotion engine to analyze the user's voice, facial expressions, and input speed to estimate their emotions. This emotion data is also recorded as part of the operation log.

[0695] Next, the server receives emotional data along with the operation logs acquired from the terminal. The received data is analyzed using a generative AI, and the recurring patterns hidden in the user's actions are cross-referenced and analyzed with the emotions at that time. Based on this analysis, the server understands what emotional states the user is most likely to experience in what situations, and uses that information to generate an automated script.

[0696] The generated automation scripts dynamically change their actions based on the user's emotional state. For example, they are designed to execute instructions that simplify tasks when the user is frustrated, and to perform the normal process when the user is calm.

[0697] This system can be adjusted to skip steps that users often find tedious, such as those involved in regular software updates. These adjustments, based on emotional data, allow users to use the system in a more comfortable environment.

[0698] Furthermore, the server collects emotionally charged feedback from users and uses it to improve the scripts. This allows the system to continuously provide more appropriate behavior that is tailored to the user's profile.

[0699] This configuration is expected to not only reduce the user's operational burden, but also significantly improve the user experience and enhance the efficiency of system operations.

[0700] The following describes the processing flow.

[0701] Step 1:

[0702] The terminal monitors user actions on the system in real time and obtains detailed operation logs. This includes keyboard input, mouse operations, and application usage history. All operations are recorded with timestamps.

[0703] Step 2:

[0704] The device analyzes the user's facial expressions and voice tone using an emotion engine to estimate the user's emotions in real time. This emotion data is recorded along with the user's actions at that time, and identifies the user's stress level, satisfaction level, and other factors.

[0705] Step 3:

[0706] The device sends recorded operation logs and emotion data to the server at regular intervals. This transmission is securely performed using an encryption protocol, ensuring data protection.

[0707] Step 4:

[0708] The server analyzes the received operation logs and sentiment data. Using generative AI, it identifies repetitive operation patterns and analyzes how they affected the user's emotions. This analysis helps to find the optimal operation flow for the user.

[0709] Step 5:

[0710] The server generates an automated script based on the analysis results. This script is designed to dynamically change in response to the user's emotional state. For example, it may include controls to skip operations that the user might find stressful.

[0711] Step 6:

[0712] The server executes the generated script. During execution, the script monitors the user's emotional state in real time and adjusts the procedure accordingly. For example, if the user is impatient, the script simplifies the task and reduces the burden on the user.

[0713] Step 7:

[0714] The server re-evaluates the script execution results and user sentiment data, and notifies the user. This notification is provided as a report that includes details about the success or failure of the operation and any changes in sentiment.

[0715] Step 8:

[0716] Users provide feedback on script execution and notification results. This feedback includes evaluations of emotional responses and opinions on the script's effectiveness.

[0717] Step 9:

[0718] The server analyzes the collected feedback and further improves the script. It optimizes the script's content and execution timing to enhance the user experience.

[0719] This entire process results in a flexible and efficient automated system that takes user emotions and behavioral patterns into account.

[0720] (Example 2)

[0721] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0722] Traditional automation systems aim to improve user work efficiency, but often fail to consider the user's emotional state. This creates a problem where there's no way to quickly improve the user experience when they feel stressed or find the work cumbersome. As a result, the user experience suffers, and the system's efficiency cannot be fully utilized.

[0723] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0724] In this invention, the server includes means for analyzing the user's emotions and adding the analysis results to the operation log, means for dynamically adjusting the automation script based on the emotion data, and means for collecting user feedback and improving the automation script. This makes it possible to provide an appropriate automation process according to the user's emotional state.

[0725] An "operation log" refers to data that records all operations performed by a user, including all interactions within the user interface.

[0726] A "repetitive work pattern" refers to a specific process or sequence of recurring tasks identified from a user's operation history.

[0727] An "automation script" refers to a set of instructions or code used to automatically execute specific operations or processes.

[0728] "Notification" refers to the act or means by which a system communicates the results or status of an operation to a user as information.

[0729] "Emotional data" refers to information that quantifies and categorizes a user's emotional state, and is generated based on factors such as voice, facial expressions, and input speed.

[0730] "Dynamic adjustment" refers to changing content or processes in real time according to the situation.

[0731] "Feedback" refers to the opinions and reactions that users provide regarding system operation and results, and the information that can be used to improve the system in the future.

[0732] This invention provides a system that automates user operations and optimizes the automation process by taking into account the user's emotional state. The system consists of a terminal, a server, and the user who utilizes it.

[0733] The terminal is specifically designed to record user actions in real time. Specifically, it meticulously records all operations on the user interface, including CLI commands and GUI interactions. It also collects data such as the user's voice, facial expressions, and input speed, which are analyzed by a built-in emotion engine to generate emotion data. This emotion data is added to the operation log and sent to the server as a comprehensive dataset.

[0734] The server is responsible for analyzing operation logs and emotion data sent from the terminal. This analysis utilizes a generative AI model, cross-referencing repetitive patterns and emotional states hidden within the operations. This identifies work patterns associated with the user's specific emotional state. Based on this identified information, the server inputs prompts into the generative AI model, generating automated scripts to enable the user to proceed with their tasks in the desired state.

[0735] The generated automation scripts have the characteristic of dynamically adjusting their execution based on the user's emotional state. For example, if the emotional data indicates that the user is irritated, the scripts are designed to include instructions that simplify the task. This can improve the user experience. Specifically, in routine system update tasks, time-consuming steps may be skipped if the user is bored.

[0736] Furthermore, the server collects emotionally charged feedback from users and uses it as data to continuously improve automated scripts. This self-improvement process ensures that the system is always tailored to the user.

[0737] An example of a prompt is, "Suggest an automated script to shorten the steps a user has to take when they are bored." This allows the generative AI model to provide specific scripts to improve the user experience.

[0738] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0739] Step 1:

[0740] The terminal records user actions in real time. First, the terminal meticulously logs all operations performed on the user interface, such as CLI commands and GUI clicks. The input is the user's actions themselves, and the output is the recorded operation log. Through this process, the terminal maintains a complete record of all user actions.

[0741] Step 2:

[0742] The device generates emotional data. The device collects the user's voice, facial expressions, and input speed, and analyzes them using an emotion engine. The input is the collected voice and facial expression data, and the output is the analyzed emotional data. This analysis process makes it possible to convert the user's emotional state into numerical values ​​and categories.

[0743] Step 3:

[0744] The terminal sends recorded operation logs and sentiment data to the server. The terminal encrypts this data and sends it to the server using the HTTP protocol. The input is operation logs and sentiment data, and the output is the data that has been securely delivered to the server. Through this transmission, all the data necessary for analysis is aggregated on the server.

[0745] Step 4:

[0746] The server analyzes the data it receives. Using a generative AI model, the server cross-references repetitive work patterns and emotional states from operation logs and emotion data. The input is the operation logs and emotion data received by the server, and the output is the relationship between identified work patterns and emotions. This analysis allows for an understanding of which emotional states are associated with specific operations.

[0747] Step 5:

[0748] The server generates an automated script. Based on the analysis results, the server inputs prompt messages into an AI model that generates an automated script tailored to the user's emotions. The input is the analyzed work pattern information, and the output is the generated automated script. For example, a prompt message such as "When the user is frustrated, suggest a script that shortens the work procedure" might be input.

[0749] Step 6:

[0750] The server executes the generated automation script and notifies the user of the results. As it runs, the script dynamically adjusts to the user's current emotional state. The input is the generated automation script, and the output is a report of the results of the performed task. For example, if the user is feeling stressed, the task steps will be simplified.

[0751] Step 7:

[0752] The user provides feedback on the system's execution results. This feedback includes information about whether the user's expectations were met and how user-friendly the system was. The input is the feedback provided by the user, and the output is a record of that feedback. This allows the system to continuously learn the user's preferences.

[0753] Step 8:

[0754] The server improves the automated scripts based on user feedback. The analyzed feedback data is used to readjust the generated AI model, modifying future scripts for better performance. The input is the feedback data, and the output is the improved automated script. This allows the system to provide users with more optimized operations over time.

[0755] (Application Example 2)

[0756] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0757] Traditional automation systems aim to reduce the user's workload, but they have limitations in improving the user experience because they do not take into account the user's emotional state. Furthermore, they lack the ability to dynamically adjust tasks based on emotions, resulting in insufficient optimization for individual users.

[0758] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0759] In this invention, the server includes means for recording user actions, means for analyzing facial expressions and voice to estimate user emotions, and means for detecting repetitive work patterns from recorded actions and emotional states. This makes it possible to improve the user experience by dynamically changing automated scripts based on the user's emotional state.

[0760] "Means for recording user actions" refers to a device or software that has the function of collecting all user actions performed on an interface as logs and storing them in a database.

[0761] "Means for detecting repetitive work patterns" refer to algorithms or programs that analyze recorded operation logs and identify patterns in which the same or similar operations are repeated.

[0762] "Means for generating automation scripts" refers to software or a system that generates instructions or code for automating tasks based on detected work patterns.

[0763] "Means of notifying information providers" refers to devices or methods that provide screen displays or audio output to directly communicate the results of the generated automation script execution to the user.

[0764] "Means for collecting user feedback and improving automation scripts" refers to a program or system that includes a process of analyzing opinions and feedback received from users and making more adaptive changes to automation scripts based on this analysis.

[0765] "Means for analyzing facial expressions and voice to estimate user emotions" refers to sensing technologies and analysis algorithms that analyze user facial expression data and voice signals to estimate their emotional state.

[0766] "Means for adjusting task content" refers to systems or programs that dynamically change automated work content according to the user's emotional state and execution environment, providing the most suitable work content.

[0767] In the system implementing this invention, the terminal first records all of the user's operations. These operations are saved in a log as CLI commands and GUI interactions. The terminal also uses software libraries such as OpenCV and DeepFace to analyze the user's facial expressions and voice data and estimate their emotions. The analysis results are recorded in a database along with the operation log and sent to the server.

[0768] Based on the received data, the server uses a generative AI model to analyze repetitive work patterns and generates automated scripts based on this information. The generated scripts are dynamically modified using Python and Node-RED, taking into account the user's emotional state. Specifically, if a user is estimated to be irritated, the system simplifies the task; conversely, if the user is judged to be focused, it performs a normal or detailed task.

[0769] Furthermore, the system notifies users of the execution results through information dissemination channels and collects feedback. This allows the automated scripts to be continuously improved based on user feedback.

[0770] For example, if a user is concerned about a specific household task (such as cleaning or cooking), they can guide the AI ​​with a prompt like this: "Generate ways to streamline household tasks while the user is feeling stressed. Infer how he needs support and recommend tasks that the robot can perform."

[0771] In this way, this invention makes it possible to provide a dynamic and efficient work environment that takes into account the user's emotional state.

[0772] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0773] Step 1:

[0774] The terminal records all operations performed by the user on the interface. Inputs include CLI commands and GUI interactions; this data is collected and stored as an operation log to prepare for subsequent processing.

[0775] Step 2:

[0776] The device analyzes the user's facial expressions and voice using software libraries such as OpenCV and DeepFace. It acquires real-time facial image and voice data from the user as input, analyzes this data to estimate the user's emotional state, and generates emotion data.

[0777] Step 3:

[0778] The terminal saves recorded operation logs and generated emotion data to a database and sends them to the server. The input consists of already generated operation logs and emotion data, which are then transferred to the server.

[0779] Step 4:

[0780] The server receives the operation logs and sentiment data and analyzes the data using a generative AI model. The input data consists of operation logs and sentiment information, which are used to detect repetitive work patterns and clarify the nature of the work.

[0781] Step 5:

[0782] The server generates automation scripts using Python scripts and Node-RED based on detected work patterns and emotional information. This creates dynamic script content that responds to the user's emotional state and outputs an appropriate execution plan.

[0783] Step 6:

[0784] The server executes the generated automation script and notifies the user of the results. The input is the generated execution plan, which is analyzed, and the results are communicated to the user in a format appropriate to the medium.

[0785] Step 7:

[0786] Users submit their thoughts and opinions to the server through the feedback form they receive. The server collects this feedback, re-enters it into the generating AI model, and uses it to improve future scripts.

[0787] This dynamically creates an efficient work environment that takes the user's emotional state into consideration.

[0788] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0789] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0790] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0791] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0792] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0793] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0794] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0795] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0796] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0797] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0798] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0799] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0800] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0801] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0802] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0803] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0804] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0805] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0806] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0807] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0808] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0809] The following is further disclosed regarding the embodiments described above.

[0810] (Claim 1)

[0811] A means for recording user actions,

[0812] A means for detecting repetitive work patterns from recorded operations,

[0813] A means for generating an automation script based on detected work patterns,

[0814] A means of executing the generated automation script and notifying the user of the results,

[0815] A means of collecting user feedback and improving automation scripts,

[0816] A system that includes this.

[0817] (Claim 2)

[0818] The system according to claim 1, comprising means for recording user operations in real time and transmitting the log data to a server.

[0819] (Claim 3)

[0820] The system according to claim 1, comprising means for optimizing the execution efficiency of the generated automation script.

[0821] "Example 1"

[0822] (Claim 1)

[0823] A means of using an information terminal to monitor operations and acquire data,

[0824] A communication means for transferring the acquired data to the analysis device,

[0825] A data analysis device provides means for identifying repetitive tasks and constructing automated procedures using a generative AI model,

[0826] A means for executing a constructed automated procedure and reporting the results to the user via a display device,

[0827] A means of improving the automated procedure based on information received from users,

[0828] A system that includes this.

[0829] (Claim 2)

[0830] The system according to claim 1, comprising a function to immediately record user activity and transmit this record to an analysis device.

[0831] (Claim 3)

[0832] The system according to claim 1, comprising a method for improving the operational efficiency of the generated automated procedures.

[0833] "Application Example 1"

[0834] (Claim 1)

[0835] A means for recording the actions performed by a terminal device,

[0836] A means for detecting periodic motion patterns from recorded motion,

[0837] Means for generating automated control procedures based on detected operating patterns,

[0838] A means of executing the generated automated control procedure and notifying a human of the report,

[0839] A means for recording and transmitting feedback and improving automated control procedures,

[0840] Means for optimizing the work efficiency of robotic devices,

[0841] A system that includes this.

[0842] (Claim 2)

[0843] The system according to claim 1, comprising means for recording the operation of a terminal device in real time and transmitting the log information to a storage device.

[0844] (Claim 3)

[0845] The system according to claim 1, comprising means for improving the execution efficiency of the generated automated control procedure.

[0846] "Example 2 of combining an emotion engine"

[0847] (Claim 1)

[0848] A means for recording user actions,

[0849] A means for detecting repetitive work patterns from recorded operations,

[0850] A means for generating an automation script based on detected work patterns,

[0851] A means of executing the generated automation script and notifying the user of the results,

[0852] A means of analyzing user emotions and adding the analysis results to the operation log,

[0853] A means of dynamically adjusting automated scripts based on emotional data,

[0854] A means of collecting user feedback and improving automation scripts,

[0855] A system that includes this.

[0856] (Claim 2)

[0857] The system according to claim 1, comprising means for recording user operations in real time and transmitting the log data to a server.

[0858] (Claim 3)

[0859] The system according to claim 1, comprising means for optimizing the execution efficiency of the generated automation script.

[0860] "Application example 2 when combining with an emotional engine"

[0861] (Claim 1)

[0862] A means for recording user actions,

[0863] A means for detecting repetitive work patterns from recorded operations,

[0864] A means for generating an automation script based on detected work patterns,

[0865] The generated automation script is executed while dynamically changing it according to the user's emotional state.

[0866] Means of notifying the results to information dissemination media,

[0867] A means of collecting user feedback and improving automation scripts,

[0868] A system that includes means of analyzing facial expressions and voice to estimate the user's emotions.

[0869] (Claim 2)

[0870] The system according to claim 1, comprising means for recording user operations in real time and transmitting the log data to a processing device.

[0871] (Claim 3)

[0872] The system according to claim 1, comprising means for optimizing the execution efficiency of the generated automation script while adjusting the task content based on the user's emotional state. [Explanation of symbols]

[0873] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for recording user actions, A means for detecting repetitive work patterns from recorded operations, A means for generating an automation script based on detected work patterns, A means of executing the generated automation script and notifying the user of the results, A means of collecting user feedback and improving automation scripts, A system that includes this.

2. The system according to claim 1, comprising means for recording user operations in real time and transmitting the log data to a server.

3. The system according to claim 1, further comprising means for optimizing the execution efficiency of the generated automation script.

Images