system

Abstract

Provide a system. A means for recording user operations; means for analyzing the recorded operations and identifying repetitive tasks; a means for suggesting automation measures based on the identified repetitive tasks; means for implementing the proposed automation means on a user's terminal; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In many companies, employees spend a lot of time on repetitive, routine tasks, leaving them with insufficient time to focus on creative or strategic tasks. Furthermore, existing business efficiency tools only automate a portion of tasks, often overlooking the overall efficiency of the workflow. Implementing these efficiency tools also requires specific business knowledge and skills, making them complex. The objective of this invention is to provide a system that solves these problems and achieves business efficiency. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for recording user operations, a means for analyzing the recorded operations and identifying repetitive tasks, a means for proposing automation measures based on the identified repetitive tasks, and a means for implementing the proposed automation measures on the user's terminal. The system also includes a means for saving data on the recorded operations and a means for learning operation patterns based on the saved data, enabling the system to accurately analyze the user's workflow and propose and execute automation measures. Furthermore, the system adds a means for evaluating the frequency of execution of identified tasks and determining automation priorities based on that frequency, enabling effective automation. This configuration automates repetitive routine tasks performed by employees, providing an environment in which employees can focus on creative work.

[0006] The "means for recording operations" is a hardware or software mechanism for detecting operations performed by a user on a computer and recording the details of those operations as data.

[0007] The "means for analyzing operations and identifying repetitive tasks" has the function of analyzing recorded operation data and identifying repetitive tasks based on specific patterns or frequencies.

[0008] A "means for suggesting automation methods" is a function that suggests appropriate automation methods (e.g., scripts or RPA tools) for identified repetitive tasks.

[0009] "Means for implementation on user devices" refers to the mechanisms for configuring and deploying the proposed automation measures so that they function on the computers or devices actually used by users.

[0010] "Means for storing recorded operation data" refers to a database or storage system for safely and effectively storing and managing user operation records.

[0011] The "means for learning operation patterns" refers to a function that uses a machine learning algorithm to learn user operation patterns based on saved operation data and use the learned patterns to make future predictions and suggestions.

[0012] The "means for evaluating the frequency of task execution and determining the priority of automation" is a means for evaluating how frequently identified tasks are executed and, based on that evaluation, determining which tasks should be prioritized for automation. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] System Overview

[0035] This invention is a system that automatically identifies repetitive tasks by recording and analyzing user operations, and proposes and implements appropriate automation measures. This system involves a series of processes: installing client software on the user's device, and analyzing and saving operation data on a server.

[0036] Program processing

[0037] In the embodiment of the present invention, the following program processing is performed.

[0038] 1. Operation Record

[0039] Device:

[0040] Client software installed on the user's PC records user actions in real time, including clicks, keyboard input, application startup and shutdown, file operations, etc. For example, a user entering data into a spreadsheet is recorded.

[0041] 2. Data transmission and storage

[0042] Device:

[0043] The recorded operation data is periodically sent to a server, including the date, time, and details of the operation.

[0044] server:

[0045] The received data is stored in a database, allowing all operation histories to be managed together and used for later analysis.

[0046] 3. Data analysis and identification of repetitive tasks

[0047] server:

[0048] The stored operational data is analyzed using machine learning algorithms to identify specific operational patterns. Frequently performed operations or tasks that are repeated based on certain rules are identified as repetitive tasks. For example, entering data into a spreadsheet in the same format every day can be identified.

[0049] 4. Proposal of automation methods

[0050] Generated AI (server):

[0051] Based on the identified repetitive tasks, it recommends the best means of automation, which can include generating scripts or using RPA tools. For example, to automate data entry into a spreadsheet, it suggests a data entry script that follows a specific format.

[0052] 5. Creating and implementing automation tools

[0053] server:

[0054] Scripts and configuration files are generated to implement the proposed automation methods, and the settings are configured to be deployed to the user's device.

[0055] Device:

[0056] Install the generated scripts and tools and execute the automation according to your settings. When a user opens a spreadsheet, the configured scripts will automatically populate the data.

[0057] 6. Implementing automation and improving business efficiency

[0058] User:

[0059] Ensure your automation tools are working properly and, if necessary, learn how to use them, which will free up time to focus on creative and strategic tasks rather than manual, repetitive tasks.

[0060] Specific examples

[0061] Example 1: Creating a market analysis report

[0062] Device:

[0063] The user's actions of periodically collecting market data and creating reports in Excel are recorded.

[0064] server:

[0065] Analyze recorded operational data to identify regular and consistent patterns of reporting.

[0066] Generated AI (server):

[0067] Propose a script to automate the collection of market data and inputting it into Excel.

[0068] server:

[0069] Generate the proposed script and deploy it to the user's device.

[0070] Device:

[0071] When the user runs the script, market data is automatically retrieved and an Excel report is automatically created.

[0072] User:

[0073] Time spent on report generation is reduced and can be used for data analysis and decision making.

[0074] Example 2: Managing meeting schedules

[0075] Device:

[0076] It records actions taken by users when they schedule regular meetings, such as setting up a meeting using Outlook Calendar and inviting attendees.

[0077] server:

[0078] As a result of analyzing the operational data, scheduling meetings is identified as a recurring task that is performed periodically.

[0079] Generated AI (server):

[0080] Suggest a macro or script to automate meeting scheduling in Outlook.

[0081] server:

[0082] Generate suggested macros and scripts and prepare them for deployment to users' devices.

[0083] Device:

[0084] Macros and scripts automatically schedule meetings and invite attendees.

[0085] User:

[0086] Free yourself from manually scheduling meetings and use that time to coordinate and execute important projects.

[0087] This frees users from repetitive routine tasks, allowing them to focus on more advanced tasks. The system generates AI specialized for specific business processes, improving business efficiency and optimization.

[0088] The processing flow will be explained below.

[0089] Step 1:

[0090] Device:

[0091] When a user starts working on their PC, the client software automatically launches in the background. The client software records operation data in real time, including clicks, keystrokes, application launches and exits, and file operations. This operation data includes the date and time of the operation, as well as details of the application and file involved.

[0092] Step 2:

[0093] Device:

[0094] The recorded operation data is sent to the server at regular intervals. The frequency of data transmission can be adjusted based on network load and security considerations. For example, it can be set to send operation data in batch processing every hour.

[0095] Step 3:

[0096] server:

[0097] The operation data received by the server is stored in a database in real time. The database stores the operation history for each user in chronological order and is used for later analysis. The stored data includes detailed information about the operation and the application in question.

[0098] Step 4:

[0099] server:

[0100] The stored operation data is analyzed using a machine learning algorithm. The algorithm extracts frequently performed operation patterns from the operation data and identifies tasks that are repeated according to specific rules. For example, an operation pattern in which a user enters data into a spreadsheet at the same time every day can be identified.

[0101] Step 5:

[0102] Generated AI (server):

[0103] Based on the identified repetitive tasks, the generative AI suggests the best way to automate them. These suggestions may include generating scripts or using RPA (Robotic Process Automation) tools. The suggestions are customized for each user, showing the most efficient method.

[0104] Step 6:

[0105] server:

[0106] Based on the proposed automation method, specific scripts and configuration files are generated, which contain detailed instructions for the tasks to be automated, and are then prepared for deployment to the user's devices.

[0107] Step 7:

[0108] Device:

[0109] Receive deployed scripts and automation tools from the server, install and configure them according to the configuration files, test the automation tools to ensure they are working properly, and make adjustments as needed.

[0110] Step 8:

[0111] User:

[0112] Execute the automated task and see the results. For example, by automating data entry into a spreadsheet, a task that was previously done manually can now be done instantly by a script. Users can then use the time saved by automation to focus on other creative tasks.

[0113] Step 9:

[0114] server:

[0115] Regularly monitor the use of automation tools and evaluate their effectiveness, checking whether users are becoming more efficient and making additional suggestions or adjustments if further improvements are needed.

[0116] Through the above series of processing steps, users are freed from repetitive routine tasks and can concentrate on creative work.

[0117] Example 1

[0118] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0119] In modern work environments, there are many repetitive tasks that users perform repeatedly. However, these tasks are often performed manually, significantly reducing efficiency. It is also difficult for users to determine which tasks should be automated, and selecting and implementing appropriate automation tools is time-consuming. Therefore, there is a need for a system that records user operations, automatically identifies repetitive tasks, and proposes and implements the optimal automation methods.

[0120] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0121] In this invention, the server includes means for recording user operations in real time, means for periodically transmitting the recorded operation data to the server, means for storing the received data in a database, means for analyzing the stored data using a machine learning algorithm to identify repetitive tasks, means for proposing appropriate automation measures based on the identified repetitive tasks, means for generating scripts or tools that implement the proposed automation measures and installing them on the user's terminal, and means for verifying that the automation tools operate normally on the user's terminal. This frees users from repetitive tasks, allowing them to use their time efficiently and focus on more valuable tasks.

[0122] "User operations" refer to actions such as clicking, keyboard input, starting and closing applications, and file operations that a user performs on a device.

[0123] "Real-time recording means" refers to software or a system that instantly records user operations and saves the data.

[0124] "Operation data" is digital data that includes the content, date and time, details of a user's operation, and information about the application being used.

[0125] The "means for transmitting to the server" refers to a communication protocol or program for transmitting the operation data recorded on the terminal to the server.

[0126] The "means for storing in a database" refers to a system or method for storing the received operation data in a database in a form that allows it to be uniquely identified.

[0127] A "machine learning algorithm" is an algorithm or model that analyzes stored operation data and identifies specific operation patterns.

[0128] A "repetitive task" is an operation or task that is performed in the same sequence and is repeated at regular intervals.

[0129] An "automation tool" is a tool, software, script, or system used to perform repetitive tasks automatically.

[0130] A "script or tool generating means" is a system or software for generating code or configuration files to implement the proposed automation means.

[0131] The "means for introducing the script or tool into the user's terminal" refers to a method or process for placing the generated script or tool on the user's terminal and running it properly.

[0132] "Means to verify proper operation" refers to the process or method for verifying that the implemented scripts and tools operate as intended.

[0133] MODE FOR CARRYING OUT THE INVENTION

[0134] The present invention is a system that automatically identifies repetitive tasks by recording and analyzing user operations, and then proposes and implements appropriate automation measures. This system installs client software on the user's terminal, and analyzes and stores data on a server. Specific embodiments of the system are described below.

[0135] System Overview

[0136] The system records user operations in real time, transmits the operation data to a server, and stores the received data in a database. It uses a machine learning algorithm to analyze operation patterns and identify repetitive tasks. Based on the identified tasks, it proposes automation methods and includes a means for implementing them on the user's device.

[0137] Hardware and software used

[0138] Device:

[0139] Client software is installed on the user's PC, and this software has the function of recording user operations in real time and periodically sending the data to the server.

[0140] server:

[0141] The server stores the received operation data in a database and analyzes the data using machine learning algorithms. For example, it uses a machine learning library such as TENSORFLOW (registered trademark). Based on the analysis results, it proposes automation methods, generates scripts and tools, and deploys them to the user's device.

[0142] Database:

[0143] The database stores operational data in a uniquely identifiable form, which is then used for later analysis by machine learning algorithms.

[0144] Specific actions

[0145] User operation record:

[0146] The client software on the device records user actions such as clicks, keyboard input, and application launches and exits, generating detailed operational data.

[0147] Data transmission and storage:

[0148] The recorded operation data is sent to a server, for example, every five minutes, and the server stores the received data in a database for later analysis.

[0149] Data analysis and automation methods proposed:

[0150] The server analyzes the stored data using machine learning algorithms to identify repetitive tasks, and then suggests the best automation method (script or tool) based on the identified tasks.

[0151] Generate and deploy scripts and tools:

[0152] The proposed automation methods are embodied as scripts or tools and deployed to the user's terminals, for example, Python scripts or RPA tools.

[0153] Check the operation of the automation tool:

[0154] Users ensure that automated scripts and tools work properly, automating repetitive tasks and improving work efficiency.

[0155] Specific examples

[0156] Example 1: Creating a market analysis report

[0157] Device:

[0158] This tool records the actions of users who periodically collect market data and create reports in Excel. It also records the actions of downloading data from market data sites and entering it into Excel sheets.

[0159] server:

[0160] The recorded operation data is analyzed and "downloading market data" and "entering it into an Excel spreadsheet" are identified as repetitive tasks.

[0161] Generated AI (server):

[0162] We propose a Python script to automate the collection of market data and inputting it into Excel.

[0163] server:

[0164] Generates the proposed Python script and deploys it to the user's device.

[0165] Device:

[0166] When the user runs the script, market data is automatically retrieved and an Excel report is automatically generated.

[0167] User:

[0168] Spend less time creating reports and more time analyzing data and making decisions.

[0169] Prompt Sentence Examples

[0170] "Please suggest a script to record and automate the regular market data collection and Excel report creation."

[0171] Example 2: Managing meeting schedules

[0172] Device:

[0173] This records the actions of users who schedule regular meetings, set up meetings in their Outlook calendar, and invite attendees.

[0174] server:

[0175] By analyzing the operation data, "scheduling a meeting" and "inviting attendees" are identified as repetitive tasks.

[0176] Generated AI (server):

[0177] Suggest a VBA macro or Python script to automate meeting scheduling in Outlook.

[0178] server:

[0179] Generate suggested macros and scripts and prepare them for deployment to users' devices.

[0180] Device:

[0181] Macros and scripts automatically schedule meetings and invite attendees.

[0182] User:

[0183] Free yourself from the manual work of scheduling meetings and use that time to coordinate and execute important projects.

[0184] Prompt Sentence Examples

[0185] "Please suggest a macro or script that records and automates routine meeting setup and attendee invitation actions."

[0186] This frees users from repetitive tasks, allowing them to focus on more creative and strategic work.

[0187] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0188] Step 1:

[0189] Operation Record

[0190] Device:

[0191] Client software installed on the user's PC records the user's actions in real time. Specifically, each time the user clicks, the location information, the details of the action (e.g., button click, form entry), and the application being used are captured. For example, the user entering "sales data" and setting a filter in a spreadsheet is recorded.

[0192] input:

[0193] User actions (clicks, keyboard input, etc.)

[0194] output:

[0195] Operation data (location information, action details, application information, etc.)

[0196] Step 2:

[0197] Data transmission

[0198] Device:

[0199] At regular intervals (for example, every 5 minutes), the recorded operation data is sent to the server. The sent data includes the date and time of the operation, detailed content, and information about the application used.

[0200] input:

[0201] Recorded operation data

[0202] output:

[0203] Operation data sent to the server

[0204] Step 3:

[0205] Data storage

[0206] server:

[0207] The received data is saved in a database. An ID and timestamp are assigned to the database so that operation history can be stored in a uniquely identifiable form. For example, the operation "Enter sales data into a spreadsheet" performed on "2023-01-01 12:00:00" is saved as "Operation History ID: 12345."

[0208] input:

[0209] Operation data sent to the server

[0210] output:

[0211] Operational data stored in a database

[0212] Step 4:

[0213] Data analysis

[0214] server:

[0215] The stored data is analyzed using machine learning algorithms. For example, machine learning libraries such as TensorFlow are used. Through the analysis, frequently performed operations or tasks that are repeated based on certain rules are identified as "repetitive tasks." For example, it may be detected that "entering sales data into a spreadsheet" is performed every day.

[0216] input:

[0217] Operational data stored in a database

[0218] output:

[0219] Repetitive task identification results

[0220] Step 5:

[0221] Proposal of automation methods

[0222] Generated AI (server):

[0223] Based on the analysis results, the system proposes optimal automation methods for identified repetitive tasks. Specifically, it generates scripts or bots (e.g., Python scripts or RPA tools) that mimic the operational procedures and suggests them to the user. For example, it proposes generating a Python script that automatically inputs sales data into a spreadsheet.

[0224] input:

[0225] Repetitive task identification results

[0226] output:

[0227] Proposal of automation methods (scripts and tool specifications)

[0228] Step 6:

[0229] Generating scripts and tools

[0230] server:

[0231] Generate scripts and configuration files to implement the proposed automation measures, for example, generate the proposed Python scripts and package them into a ZIP file.

[0232] input:

[0233] Proposal of automation methods (scripts and tool specifications)

[0234] output:

[0235] Concretized scripts and tools (e.g. generated Python scripts)

[0236] Step 7:

[0237] Deploying scripts and tools

[0238] server:

[0239] Prepare the generated scripts and tools for deployment to the user's device and perform the necessary configuration.

[0240] Device:

[0241] Install the generated scripts and tools on the terminal and register them in the task scheduler to run at 10:00 every day.

[0242] input:

[0243] Embedding scripts and tools (ZIP files, etc.)

[0244] output:

[0245] Scripts and tools deployed on user devices

[0246] Step 8:

[0247] Operation check

[0248] User:

[0249] Ensure automated scripts and tools are working properly and, if necessary, learn how to use them. Well-working scripts automate repetitive tasks and improve work efficiency.

[0250] input:

[0251] Scripts and tools deployed on user devices

[0252] output:

[0253] Confirmed working automation tools

[0254] (Application example 1)

[0255] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0256] The large amount of repetitive manual work at logistics centers consumes a lot of labor and time, resulting in issues such as reduced productivity and increased human error. In particular, efficient automation is required for routine tasks such as picking and inventory taking. Automating these tasks will improve operational efficiency and reduce the burden on workers.

[0257] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0258] In this invention, the server includes means for recording user operations, means for analyzing the recorded operations and identifying repetitive tasks, means for proposing automation measures based on the identified repetitive tasks, means for installing the proposed automation measures on the user's terminal, means for transmitting and storing operation data recorded in logistics operations to a cloud server, means for analyzing the stored data using a machine learning algorithm and identifying repetitive work patterns, and means for generating automation scripts based on the identified work patterns and installing them on the user's mobile device. This makes it possible to efficiently automate repetitive routine tasks performed in logistics centers, reduce the burden on workers, and improve productivity.

[0259] "User" means a person who operates the System or an end user.

[0260] "Means for recording operations" refers to software or devices for capturing and saving user operations in real time.

[0261] A "repetitive task" is work that follows a set pattern, where the same operations are repeated over and over again.

[0262] "Automation tools" are scripts and tools proposed to automate repetitive user tasks.

[0263] "Terminal deployment" refers to the process of installing or making executable the automation means on the user's device.

[0264] A "cloud server" is an external server that stores and analyzes data via the Internet.

[0265] A "machine learning algorithm" is an algorithm that learns patterns from large amounts of data and makes predictions and classifications for new data.

[0266] A "work pattern" is a repetitive action or sequence of operations performed by a user.

[0267] An "automation script" is a program or code created to perform a specific task automatically.

[0268] A "mobile device" is a portable electronic device such as a smartphone or tablet.

[0269] "Storage means" refers to a memory or database for reliably storing recorded data.

[0270] "Proposed means" are functions or processes that show users the automated means.

[0271] "Means for analysis" refers to the systems and processes used to analyze the recorded operational data and extract meaningful information.

[0272] This invention is a system that aims to automate repetitive tasks in logistics centers and maximize operational efficiency. Specifically, it records user operations in real time and analyzes them using machine learning algorithms to identify repetitive tasks and propose and implement automation methods.

[0273] Overall system configuration

[0274] 1. A means of recording user actions

[0275] The client application installed on the device records logistics operations in real time, such as QR code scanners and touchscreen input, including clicks, keyboard input, application launches and exits, and file operations.

[0276] 2. Means of transmitting and storing recorded operating data

[0277] The operation data recorded on the device is periodically sent to a cloud server and stored in AWS (registered trademark) or a database (SQLite, PostgreSQL, etc.). This data includes the date and time of the operation and details.

[0278] 3. Data analysis and identification of repetitive tasks

[0279] The server analyzes the stored operation data using machine learning algorithms (e.g., Python, TensorFlow, etc.), thereby identifying frequently performed operations and tasks that are repeated based on certain rules.

[0280] 4. Proposal of automation methods

[0281] The server uses a generative AI model to suggest optimal automation strategies based on identified repetitive tasks, taking into account the frequency and priority of the tasks. For example, a script may be generated for a specific item picking or periodic inventory task.

[0282] 5. Creating and implementing automation tools

[0283] Based on the proposed automation measures, scripts and configuration files are generated and deployed to users' mobile devices (smartphones and tablets). The scripts automate certain repetitive tasks, such as automatically updating new inventory data.

[0284] 6. Automation and operational efficiency

[0285] Using the generated automation tools, users can streamline logistics operations. By automating picking and inventory tasks, the burden on workers is reduced, allowing them to spend more time focusing on important tasks.

[0286] Specific examples

[0287] Example 1: Automated product picking

[0288] At a logistics center, the system records the product picking work performed by workers every day. The data sent to a cloud server is analyzed and repetitive picking patterns are identified. Based on the results, a script to automate the picking work is generated and deployed to smartphones. Workers can use their smartphones to automate picking work, allowing them to work more efficiently.

[0289] Example prompts to input to a generative AI model:

[0290] Generate a script to automate the daily picking tasks performed by distribution center workers. You analyzed the following operational data and identified repetitive picking patterns. Provide an example of a script that would automate this task.

[0291] This system efficiently automates repetitive routine tasks at logistics centers, significantly reducing the burden on workers. By utilizing a combination of generative AI models and machine learning algorithms, it achieves advanced operational efficiency.

[0292] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0293] Step 1:

[0294] When working in the distribution center, users use a client application on their smartphones that uses a QR code scanner or touch screen to record operational data such as picking and inventory in real time.

[0295] Input: Product identification information and work instructions

[0296] Output: Data recording the type and time of operation

[0297] Specific operation: Scan the QR code and save the operation details and timestamp to local storage.

[0298] Step 2:

[0299] The device periodically transmits the recorded data to the cloud server, including detailed information about the operation, the date and time, and information about the working environment.

[0300] Input: Operation data stored in local storage

[0301] Output: Data transferred to the cloud server

[0302] What happens: A scheduled job collects data and sends it over the internet to a cloud server.

[0303] Step 3:

[0304] The server stores the received operational data in a database, where it is formatted and preprocessed for analysis.

[0305] Input: Operation data sent

[0306] Output: Formatted operation data stored in the database

[0307] What happens: The server's receiving module parses the data and imports it into the database in the appropriate format.

[0308] Step 4:

[0309] The server runs machine learning algorithms on the stored operational data to identify repetitive work patterns, using algorithms such as Python and TensorFlow.

[0310] Input: Operational data in the database

[0311] Output: A list of identified recurring tasks

[0312] What it does: Train and evaluate machine learning models to extract important patterns.

[0313] Step 5:

[0314] The server uses a generative AI model to suggest automation measures based on the identified repetitive tasks, and the suggested automation measures are generated in the form of scripts.

[0315] Input: A list of identified recurring tasks

[0316] Output: Automation script

[0317] Specific operation: Provide input data to the generative AI model and obtain the generated script.

[0318] Step 6:

[0319] The proposed automation script is deployed to the device, where the user receives the automation script and prepares it to run on the device.

[0320] Input: Automation script

[0321] Output: Script delivered to the terminal

[0322] Specific operation: The server sends the script to the device, and the application on the device installs or updates it.

[0323] Step 7:

[0324] Users run automation scripts on their devices, which automatically process designated tasks and re-record operation data, improving work efficiency.

[0325] Input: Installed automation script

[0326] Output: The result of the automated task

[0327] What it does: The user launches the script, and the device automatically performs the task and displays the results.

[0328] Through this series of steps, repetitive tasks within the logistics center can be automated, dramatically improving work efficiency.

[0329] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0330] System Overview

[0331] This invention achieves higher business efficiency by combining a system that automatically identifies repetitive tasks by recording and analyzing user operations and proposes and implements appropriate automation measures with an emotion engine that recognizes user emotions. This system involves installing client software and an emotion engine on the user's device and a series of processes that analyze and store operation data and emotion data on a server.

[0332] Program processing

[0333] In the embodiment of the present invention, the following program processing is performed.

[0334] 1. Operation Record and Emotion Recognition

[0335] Device:

[0336] When a user starts working on their PC, the client software and emotion engine automatically start in the background. The client software records operation data such as clicks, keystrokes, application startup / exit, and file operations in real time. At the same time, the emotion engine analyzes the user's facial expressions and tone of voice via the webcam and microphone to identify the user's emotional state.

[0337] 2. Data transmission and storage

[0338] Device:

[0339] The recorded operation data and emotion data are sent to the server at regular intervals. The frequency of data transmission can be adjusted based on network load and security considerations. For example, it can be set to send operation data and emotion data in batch processing every hour.

[0340] server:

[0341] The received operation data and emotion data are stored in a database in real time. The database stores each user's operation history and emotion history in chronological order and is used for later analysis. The stored data includes details of the operation, the target application, and the user's emotional state.

[0342] 3. Data analysis and identification of repetitive tasks

[0343] server:

[0344] The system analyzes the saved operation data using a machine learning algorithm to identify specific operation patterns. Frequently performed operations and tasks that are repeated according to certain rules are identified as repetitive tasks. At the same time, it analyzes emotional data to identify which tasks cause stress or frustration for the user. For example, while identifying an operation pattern in which a user inputs data into a spreadsheet at the same time every day, it also captures emotional data indicating the user is frustrated while performing the task.

[0345] 4. Proposal of automation methods

[0346] Generated AI (server):

[0347] The generative AI then proposes optimal automation methods based on the identified repetitive tasks and emotional data. These proposals include script generation and the use of RPA (Robotic Process Automation) tools. Automation methods are prioritized for tasks that are likely to cause stress to users. For example, a script for automating data entry into a spreadsheet is proposed, and emotional data indicates that automation is urgently needed.

[0348] 5. Creating and implementing automation tools

[0349] server:

[0350] Based on the proposed automation method, specific scripts and configuration files are generated, which contain detailed instructions for the tasks to be automated, and are then prepared for deployment to the user's devices.

[0351] Device:

[0352] Receive deployed scripts and automation tools from the server, install and configure them according to the configuration files, test the automation tools to ensure they are working properly, and make adjustments as needed.

[0353] 6. Implementing automation and improving business efficiency

[0354] User:

[0355] Execute the automated task and observe the effect. For example, by automating data entry into a spreadsheet, a task that was previously done manually can now be done instantly by a script. Also, observe that the emotion engine reduces the user's stress level. Users can use the time saved by automation to pursue other creative tasks.

[0356] Specific examples

[0357] Example 1: Creating a market analysis report and managing stress

[0358] Device:

[0359] The system records the user's actions as they periodically collect market data and create reports in Excel. At the same time, an emotion engine detects whether the user is feeling stressed from their facial expressions while creating the reports.

[0360] server:

[0361] The recorded operation data and emotion data are analyzed to identify whether report creation is occurring regularly and in the same pattern, and to identify that the user is feeling stressed while performing the task.

[0362] Generated AI (server):

[0363] They propose a script to automate the collection of market data and inputting it into Excel, and determine that automation is urgently needed based on sentiment data.

[0364] server:

[0365] Generate the proposed script and deploy it to the user's device.

[0366] Device:

[0367] When the user runs the script, market data is automatically retrieved and an Excel report is automatically created.

[0368] User:

[0369] Time spent on report creation is reduced and can be spent on data analysis and decision-making, reducing stress and improving work efficiency.

[0370] Example 2: Meeting schedule management and emotion monitoring

[0371] Device:

[0372] The system records the user's actions to schedule regular meetings, such as setting up meetings using Outlook Calendar and inviting attendees. The emotion engine also detects frustration from the user's facial expressions during the meeting setup process.

[0373] server:

[0374] Analysis of the operation data identifies scheduling a meeting as a repetitive task that is performed periodically, and emotion data reveals that the task is stressful for the user.

[0375] Generated AI (server):

[0376] Suggest a macro or script to automate meeting scheduling in Outlook, and determine based on sentiment data that automation of this task should be a priority.

[0377] server:

[0378] Generate suggested macros and scripts and prepare them for deployment to users' devices.

[0379] Device:

[0380] Macros and scripts automatically schedule meetings and invite attendees.

[0381] User:

[0382] This frees employees from manually scheduling meetings and allows them to focus on coordinating and implementing important projects. Emotional data helps reduce stress and improve overall work efficiency.

[0383] In this way, the present invention improves the efficiency of users' work, and the emotion engine monitors the user's emotional state and prioritizes the automation of stressful tasks, allowing users to work in a more comfortable environment.

[0384] The processing flow will be explained below.

[0385] Step 1:

[0386] Device:

[0387] When a user starts working on their PC, the client software and emotion engine automatically start in the background. The client software records user operation data in real time, such as clicks, keystrokes, application launches and exits, and file operations. At the same time, the emotion engine analyzes the user's facial expressions and tone of voice via the webcam and microphone to identify the user's emotional state in real time.

[0388] Step 2:

[0389] Device:

[0390] The recorded operation data and emotion data are sent to a server at regular intervals, for example, every 30 minutes, and are encrypted to ensure data integrity.

[0391] Step 3:

[0392] server:

[0393] The received operation data and emotion data are stored in a database. The stored data includes the operation details (clicks, input, file operations, etc.) and the user's emotional state (stress, frustration, satisfaction, etc.) along with the date and time.

[0394] Step 4:

[0395] server:

[0396] The stored data is then analyzed using machine learning algorithms. Operational data is used to extract frequent patterns and identify repetitive tasks, while emotional data is used to understand the emotional state a user is experiencing while performing a particular task. For example, it can identify that a user is experiencing high levels of stress while regularly entering data into a spreadsheet.

[0397] Step 5:

[0398] Generated AI (server):

[0399] Based on the identified repetitive tasks and emotion data, the system suggests automation solutions, including script generation and the use of RPA (Robotic Process Automation) tools. Automation solutions are also prioritized for tasks that users find stressful. For example, a script to automate data entry into a spreadsheet is suggested.

[0400] Step 6:

[0401] server:

[0402] Based on the proposed automation measures, specific scripts and configuration files are generated, which include detailed instructions for executing the automation measures, and the generated scripts and configuration files are prepared for deployment to user devices.

[0403] Step 7:

[0404] Device:

[0405] Receives deployed scripts and automation tools from the server, installs them according to the configuration file, and then tests whether the automation tool works properly and adjusts the configuration if necessary.

[0406] Step 8:

[0407] User:

[0408] Run automated tasks and see the results: for example, an automated script instantly inputs data into a spreadsheet, and the emotion engine reduces stress levels.

[0409] Step 9:

[0410] server:

[0411] Regularly monitor the usage and sentiment data of automation tools to evaluate their effectiveness, confirming that automation is improving work efficiency and reducing user frustration, and make additional suggestions or adjust settings as needed.

[0412] Example 2

[0413] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0414] While conventional task automation systems were able to record and analyze user operations, it was difficult to propose automation methods that took the user's emotional state into account. As a result, it was not possible to appropriately prioritize automation for tasks that users found stressful, resulting in insufficient improvement in work efficiency.

[0415] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0416] In this invention, the server includes means for recording user operations, means for analyzing the recorded operations and identifying repetitive tasks, means for proposing automation measures based on the identified repetitive tasks and emotion data, means for installing the proposed automation measures in the user's terminal, and means for transmitting the recorded operation data and emotion data to the server at regular intervals and storing them in a database. This allows the user to appropriately set automation priorities for tasks that cause stress, thereby improving work efficiency.

[0417] "Means for recording user operations" refers to software or hardware that has the ability to record in real time clicks, keystrokes, application startup / exits, file operations, etc., performed by a user on a PC or other device.

[0418] The "means for identifying repetitive tasks" is a component that analyzes recorded user operation data and runs machine learning algorithms or other analytical techniques to identify frequently performed operations or tasks that are repeated based on certain rules.

[0419] An "emotion engine" is a piece of software or hardware that analyzes a user's facial expressions, tone of voice, etc. to identify the user's emotional state in real time. Specifically, it has the function of detecting the user's emotions using a webcam or microphone.

[0420] "Automation suggestion method" refers to an AI model or rule-based system that suggests optimal automation methods to users based on identified repetitive tasks and sentiment data, including script generation and the use of RPA tools.

[0421] "Means for introducing into the user's terminal" refers to the process for installing and configuring the proposed automation means into the user's terminal and introducing it so that it operates normally.

[0422] The "means for transmitting to a server at regular intervals" refers to a system that has the function of transmitting the recorded operation data and emotion data to a server at specific time intervals in batch processing or in real time.

[0423] The "means for saving to a database" is a component that stores the received operation data and emotion data in a database in a format that can be analyzed later. The data is saved in chronological order and used for later analysis and search.

[0424] This invention achieves higher business efficiency by combining a system that automatically identifies repetitive tasks by recording and analyzing user operations, and proposes and implements appropriate automation measures with an emotion engine that recognizes user emotions. This system involves installing client software and an emotion engine on the user's device, and a series of processes that analyze and store operation data and emotion data on a server.

[0425] A specific embodiment of the system is given below.

[0426] System Overview

[0427] The system consists of client software and an emotion engine installed on the user's device, and a server that analyzes and stores data. The client software records operation data such as clicks, keystrokes, application launches and exits, and file operations. The emotion engine analyzes the user's facial expressions and tone of voice via a webcam and microphone to identify their emotional state.

[0428] Collecting Operational and Emotional Data

[0429] User:

[0430] When a user starts working on their PC, the client software and emotion engine automatically start up in the background, recording all of the user's actions and monitoring the user's emotional state in real time.

[0431] Data transmission and storage

[0432] Device:

[0433] The recorded operation data and emotion data are sent to a server at regular intervals. For example, it is possible to set it to send data in batches every hour. This transmission uses an encrypted communication protocol to ensure the security of the data.

[0434] server:

[0435] The received operation data and emotion data are stored in a database in real time. The operation history and emotion history for each user are stored in chronological order and used for later analysis. For example, details such as what application a specific user used on what date and month, and what tasks they felt stressed during are stored.

[0436] Data analysis and identification of repetitive tasks

[0437] server:

[0438] The saved operation data is analyzed using a machine learning algorithm to identify specific operation patterns. Frequently performed operations and tasks that are repeated according to certain rules are identified as repetitive tasks. At the same time, emotional data is analyzed to identify which tasks the user is experiencing stress or dissatisfaction with. For example, if a user enters data into a spreadsheet at the same time every day, emotional data indicating stress while performing that task can be captured.

[0439] Proposal of automation methods

[0440] Generated AI (server):

[0441] The generative AI proposes optimal automation methods based on the identified repetitive tasks and emotion data. These proposals include generating scripts and using RPA (Robotic Process Automation) tools. Automation methods are prioritized for tasks that are particularly stressful for users. Specifically, if the AI ​​determines that a user is repeatedly entering the same data into Excel and is experiencing high levels of stress while doing so, it will propose a script to automate that task.

[0442] Creating and deploying automation tools

[0443] server:

[0444] According to the proposed automation method, specific scripts and configuration files are generated, including Python scripts and configuration files for RPA (Robotic Process Automation) tools, which are then ready to be deployed to the user's devices.

[0445] Device:

[0446] Receive deployed scripts and automation tools from the server, then install and configure them based on the configuration file. After installation, test the automation tools to ensure they work properly and make any necessary adjustments. For example, verify that the generated scripts automatically input data into a spreadsheet.

[0447] Implementing automation and verifying operational efficiency

[0448] User:

[0449] Execute the automated task and check the effect. For example, by automating data entry into a spreadsheet, the previously manual data entry work can be completed instantly. The emotion engine also confirms a reduction in the user's stress level. This allows the time that would have been spent on manual tasks to be used for other tasks. In particular, work can be carried out with less stress, improving overall work efficiency.

[0450] Specific examples

[0451] Example 1: Creating a market analysis report and managing stress

[0452] Device:

[0453] The system records the user's actions as they periodically collect market data and create reports in Excel, while the emotion engine detects stress from the user's facial expressions while creating the reports.

[0454] server:

[0455] The analysis identifies regular and patterned report creation tasks and the stress experienced by users during those tasks.

[0456] Generated AI (server):

[0457] He proposes a script to automate the collection of market data and inputting it into Excel, and determines that this is an urgent need.

[0458] server:

[0459] Generate the proposed script and deploy it to the user's device.

[0460] Device:

[0461] When you run the script, it will automatically retrieve market data and generate an Excel report.

[0462] User:

[0463] The time spent on creating reports is reduced, and stress is reduced. This allows the time saved to be used for other tasks.

[0464] Example 2: Meeting schedule management and emotion monitoring

[0465] Device:

[0466] The system records users' recurring meeting scheduling actions, including setting up meetings using Outlook Calendar and inviting users to meetings. The emotion engine detects when users experience frustration during the scheduling process.

[0467] server:

[0468] Scheduling regular meetings is identified as a repetitive task, and the task proves to be stressful for the user.

[0469] Generated AI (server):

[0470] Suggest a macro or script to automate meeting scheduling in Outlook and determine that automating this task should be a priority.

[0471] server:

[0472] Generate suggested macros and scripts and deploy them to the user's device.

[0473] Device:

[0474] Automated macros and scripts schedule meetings and automatically invite people.

[0475] User:

[0476] Free yourself from manually scheduling meetings, reducing stress and freeing up time for more important work.

[0477] In this way, the present invention improves the efficiency of users' work and realizes the preferential automation of stressful tasks based on the emotion engine.

[0478] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0479] Step 1:

[0480] User:

[0481] The user starts working on the PC.

[0482] Device:

[0483] When a user logs in to their PC and begins work, the client software and emotion engine automatically start in the background. The client software records the user's clicks, keystrokes, application launches and exits, file operations, and other operational data. At the same time, the emotion engine uses the webcam and microphone to analyze the user's facial expressions and tone of voice, identifying and recording their emotional state in real time.

[0484] Input: User operations, facial expressions, and voice data at the start of the task

[0485] Output: Recorded operation data and emotion data

[0486] Step 2:

[0487] Device:

[0488] The recorded operation data and emotion data are sent to a server at regular intervals using an encrypted communication protocol to ensure data security. For example, the data is sent in batches every hour.

[0489] Input: Recorded operation data and emotion data

[0490] Output: Encrypted data sent to the server

[0491] Step 3:

[0492] server:

[0493] The server receives operation data and emotional data and stores it in a database in real time. Each user's operation history and emotional history are managed in chronological order and used for later analysis. Specific operation patterns and emotional changes can be found from the stored data.

[0494] Input: Operation data and emotion data sent to the server

[0495] Output: Operation history and emotion history stored in a database

[0496] Step 4:

[0497] server:

[0498] The saved operation data is analyzed using machine learning algorithms to identify specific operation patterns and repetitive tasks, while emotional data is analyzed to identify which tasks cause stress to the user. For example, if a user enters data into a spreadsheet at the same time every day, the system can identify a pattern of stress when doing so.

[0499] Input: Operational and emotional data stored in a database

[0500] Output: Identified repetitive tasks and emotional state data

[0501] Step 5:

[0502] Generated AI (server):

[0503] The generative AI proposes optimal automation methods based on the identified repetitive tasks and emotion data. This includes generating scripts and using RPA tools. In particular, it prioritizes automation methods for tasks that are likely to cause stress to the user. For example, if it determines that a user is repeatedly entering the same data into Excel and is experiencing high levels of stress while doing so, it proposes a script to automate that task.

[0504] Input: Identified repetitive tasks and emotional state data

[0505] Output: Proposed automation method (script, RPA tool, etc.)

[0506] Step 6:

[0507] server:

[0508] According to the proposed automation method, specific scripts and configuration files are generated, including Python scripts and configuration files for the RPA tool, which are then ready to be deployed to the user's machine.

[0509] Input: Proposed automation method

[0510] Output: Generated scripts and configuration files

[0511] Step 7:

[0512] Device:

[0513] Receive deployed scripts and automation tools from the server, install and configure them, and test the automation tools to ensure they work properly and adjust them as needed. For example, ensure that a generated script that automatically inputs data into a spreadsheet works properly.

[0514] Input: Scripts and automation tools deployed from the server

[0515] Output: Installed and configured automation tools

[0516] Step 8:

[0517] User:

[0518] Implement automated tasks and see the results. For example, automating data entry into a spreadsheet reduces manual work, and the emotion engine reduces user stress levels. This improves operational efficiency and frees up time spent on manual tasks for other important tasks.

[0519] Input: Installed automation tools

[0520] Output: The automated tasks performed and their operational efficiency effects

[0521] (Application example 2)

[0522] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0523] While conventional business efficiency systems can record user operations and automate repetitive tasks, they do not take into account the user's emotions or stress levels, which means that tasks that cause stress to the user are not automated, making it difficult to achieve effective efficiency. Furthermore, in brick-and-mortar stores, many of the tasks that employees perform daily are repetitive and their emotions fluctuate frequently, so efficiency measures that take stress levels into account are required. This leads to a decrease in employee satisfaction and work efficiency, which has a negative impact on the overall performance of the store.

[0524] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0525] In this invention, the server includes a means for recording user operations, a means for analyzing the recorded operations to identify repetitive tasks, a means for identifying the user's emotions, and a means for analyzing the emotional data and adjusting automation priorities to reduce stress. This enables automation that takes into account not only the user's operation data but also their emotional data, enabling the prioritized automation of tasks that are particularly stressful. As a result, employees are freed from stressful tasks and can focus on more creative and satisfying work, improving the operational efficiency and performance of the entire physical store.

[0526] "Means for recording user operations" refers to a function that collects and saves data such as user clicks, keystrokes, application launches and exits, and file operations in real time.

[0527] "Means for analyzing recorded operations and identifying repetitive tasks" refers to a function that analyzes collected operation data using machine learning algorithms, etc., to detect specific operation patterns that are frequently repeated.

[0528] "Means for suggesting automation measures based on identified repetitive tasks" refers to a function that recommends the use of script generation or automation tools based on data on repetitive tasks.

[0529] "Means for introducing the proposed automation measures into the user's device" refers to the ability to install the proposed automation tools or scripts on the user's device and execute the settings.

[0530] The "means for identifying user emotions" is a function that analyzes the user's facial expressions and tone of voice via a webcam or microphone to identify their emotional state.

[0531] "Means for analyzing emotional data and adjusting automation priorities to reduce stress" refers to a function that identifies tasks that cause stress to users based on emotional data, and then suggests and executes the automation of those tasks as a priority.

[0532] The "means for storing the recorded operation data" is a function for periodically sending the collected operation data to a server or database and storing it.

[0533] "Means for learning operation patterns based on stored data" refers to a function that uses stored data to utilize machine learning algorithms, etc., to identify and learn newly emerging patterns and existing patterns.

[0534] "Means for storing emotional data" refers to a function that periodically sends collected emotional data to a server or database and stores it.

[0535] The "means for analyzing stored emotion data and identifying stressful tasks" is a function for analyzing stored emotion data and identifying tasks that cause stress to the user.

[0536] The "means for evaluating the execution frequency of identified tasks and determining automation priorities based on the frequency" is a function that analyzes task execution frequency data and prioritizes automation of tasks that occur frequently.

[0537] The "means for readjusting automation priorities based on stress level" is a function that reviews the initial automation priorities based on the user's stress level data, thereby achieving more effective stress reduction.

[0538] This invention is a system for improving the work efficiency and reducing stress of employees in brick-and-mortar stores. Specific embodiments for realizing this system will be described below.

[0539] System configuration

[0540] The system includes the following hardware and software:

[0541] Smartphones or smart glasses: Equipped with a camera and microphone, they collect user interaction and emotion data.

[0542] EmotionRecognizer class: Analyzes user emotions using an open-source emotion recognition library.

[0543] TaskAutomator class: A custom library for automating tasks.

[0544] Server: Stores operation data and emotion data and performs data analysis using machine learning algorithms.

[0545] Program processing

[0546] The server first launches an application on a smartphone or smart glasses to collect user operation data in real time. The application records the employee's clicks, keystrokes, system startup and shutdown, etc. It also uses the built-in camera and microphone to collect emotional data from facial expressions and tone of voice.

[0547] The collected data is sent to a server at regular intervals, where it is stored in a database of both user operation and emotion data. The server then uses machine learning algorithms to identify repetitive tasks and analyzes the emotion data to identify tasks that cause stress to the user.

[0548] For high-stress, frequently repetitive tasks, the system uses a generative AI model to suggest optimal automation methods, which are then embodied as scripts or automation tools and deployed from a server to users' devices.

[0549] The system installs the proposed automation tool on the user's device and optimizes it based on the configuration file. During this process, the system checks the operation of the automation tool and makes adjustments as necessary.

[0550] Employees only need to perform automated tasks, significantly reducing the amount of manual work they need to do, which increases work efficiency and reduces stress levels.

[0551] Specific examples

[0552] For example, suppose there is an employee who handles accounting at a cash register in a physical store. In this case, the system operates as follows:

[0553] 1. Operation Record and Emotion Recognition

[0554] Record employees scanning items into the cash register system.

[0555] Using the camera in smart glasses or the microphone in a smartphone, it analyzes employees' facial expressions and voice to identify their emotional state.

[0556] 2. Data transmission and analysis

[0557] The collected operation data and emotion data are sent to a server.

[0558] The server stores and analyzes this data.

[0559] 3. Identifying repetitive tasks and suggesting automation

[0560] Identify that cash register operations are repetitive tasks.

[0561] Emotional data can be used to identify stress experienced while operating a cash register.

[0562] As an automation suggestion, we generate scripts to automate product scanning and accounting processes.

[0563] 4. Implementing automation tools

[0564] The scripts proposed by the server are deployed to the user's device and configured.

[0565] 5. Execute automation

[0566] Automated scripts handle cash register operations, leaving employees to simply verify the transaction.

[0567] In this way, the burden on employees can be reduced and work efficiency can be improved.

[0568] Prompt Sentence Examples

[0569] A sample prompt is as follows:

[0570] "Record cashier activity, analyze sentiment data, and suggest tasks that can be automated, such as scanning items or printing receipts."

[0571] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0572] Step 1:

[0573] When a user begins work on a device, an application installed on the smartphone or smart glasses automatically starts in the background. The application records user operation data in real time, including clicks, keystrokes, system startup / shutdown, and file operations. At the same time, it uses the built-in camera and microphone to collect emotional data from facial expressions and tone of voice. As a result, the input data becomes operation data and emotional data. The output data is temporarily stored on the device as separate datasets.

[0574] Step 2:

[0575] The device transmits the recorded operation data and emotion data to the server at regular intervals. The transmission frequency is adjusted based on network load and security considerations, for example, every 30 minutes. The input to this step is the operation data and emotion data recorded in step 1, and the output is the data transmitted to the server.

[0576] Step 3:

[0577] The operation data and emotion data received by the server are immediately stored in a database. The stored data includes a timestamp, operation details, system information of the operation target, and the user's emotional state. The input of this step is the operation data and emotion data sent from the terminal, and the output is storage in the database.

[0578] Step 4:

[0579] The server uses a machine learning algorithm to analyze the stored data and identify specific operation patterns. Repetitive operations and regularly repeated tasks are identified as repetitive tasks. At the same time, it analyzes emotional data to identify which tasks employees experience stress during. The input for this step is the operation data and emotional data stored in the database, and the output is the identified repetitive tasks and stressful tasks.

[0580] Step 5:

[0581] The server proposes automation methods based on the repetitive tasks and emotion data identified using the generative AI model. The generative AI model provides optimal methods for automating tasks, such as script generation and automation tool configuration. It also adjusts automation priorities according to the user's stress level. The inputs for this step are the repetitive tasks and stress data identified in step 4, and the output is specific automation method proposals.

[0582] Step 6:

[0583] The server instantiates the proposed automation measures and generates scripts and configuration files. The generated data is ready to be deployed to the user's device. The input to this step is the automation measures proposed in step 5, and the output is the generated scripts and configuration files.

[0584] Step 7:

[0585] The terminal receives the deployed scripts and automation tools from the server, installs and configures them on the terminal, and performs the necessary tests to verify that the tools work properly. The input to this step is the scripts and configuration files sent from the server, and the output is the installed automation tools.

[0586] Step 8:

[0587] The user executes the automated task and checks its effectiveness. For example, by automating cash register operations, employees are only required to perform the checks. This reduces work time and lowers stress levels. The input for this step is the installed automation tool, and the output is the results of the automation and the streamlined business data.

[0588] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0589] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0590] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0591] [Second embodiment]

[0592] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0593] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0594] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0595] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0596] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0597] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0598] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0599] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0600] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0601] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0602] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0603] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0604] System Overview

[0605] This invention is a system that automatically identifies repetitive tasks by recording and analyzing user operations, and proposes and implements appropriate automation measures. This system involves a series of processes: installing client software on the user's device, and analyzing and saving operation data on a server.

[0606] Program processing

[0607] In the embodiment of the present invention, the following program processing is performed.

[0608] 1. Operation Record

[0609] Device:

[0610] Client software installed on the user's PC records user actions in real time, including clicks, keyboard input, application startup and shutdown, file operations, etc. For example, a user entering data into a spreadsheet is recorded.

[0611] 2. Data transmission and storage

[0612] Device:

[0613] The recorded operation data is periodically sent to a server, including the date, time, and details of the operation.

[0614] server:

[0615] The received data is stored in a database, allowing all operation histories to be managed together and used for later analysis.

[0616] 3. Data analysis and identification of repetitive tasks

[0617] server:

[0618] The stored operational data is analyzed using machine learning algorithms to identify specific operational patterns. Frequently performed operations or tasks that are repeated based on certain rules are identified as repetitive tasks. For example, entering data into a spreadsheet in the same format every day can be identified.

[0619] 4. Proposal of automation methods

[0620] Generated AI (server):

[0621] Based on the identified repetitive tasks, it recommends the best means of automation, which can include generating scripts or using RPA tools. For example, to automate data entry into a spreadsheet, it suggests a data entry script that follows a specific format.

[0622] 5. Creating and implementing automation tools

[0623] server:

[0624] Scripts and configuration files are generated to implement the proposed automation methods, and the settings are configured to be deployed to the user's device.

[0625] Device:

[0626] Install the generated scripts and tools and execute the automation according to your settings. When a user opens a spreadsheet, the configured scripts will automatically populate the data.

[0627] 6. Implementing automation and improving business efficiency

[0628] User:

[0629] Ensure your automation tools are working properly and, if necessary, learn how to use them, which will free up time to focus on creative and strategic tasks rather than manual, repetitive tasks.

[0630] Specific examples

[0631] Example 1: Creating a market analysis report

[0632] Device:

[0633] The user's actions of periodically collecting market data and creating reports in Excel are recorded.

[0634] server:

[0635] Analyze recorded operational data to identify regular and consistent patterns of reporting.

[0636] Generated AI (server):

[0637] Propose a script to automate the collection of market data and inputting it into Excel.

[0638] server:

[0639] Generate the proposed script and deploy it to the user's device.

[0640] Device:

[0641] When the user runs the script, market data is automatically retrieved and an Excel report is automatically created.

[0642] User:

[0643] Time spent on report generation is reduced and can be used for data analysis and decision making.

[0644] Example 2: Managing meeting schedules

[0645] Device:

[0646] It records actions taken by users when they schedule regular meetings, such as setting up a meeting using Outlook Calendar and inviting attendees.

[0647] server:

[0648] As a result of analyzing the operational data, scheduling meetings is identified as a recurring task that is performed periodically.

[0649] Generated AI (server):

[0650] Suggest a macro or script to automate meeting scheduling in Outlook.

[0651] server:

[0652] Generate suggested macros and scripts and prepare them for deployment to users' devices.

[0653] Device:

[0654] Macros and scripts automatically schedule meetings and invite attendees.

[0655] User:

[0656] Free yourself from manually scheduling meetings and use that time to coordinate and execute important projects.

[0657] This frees users from repetitive routine tasks, allowing them to focus on more advanced tasks. The system generates AI specialized for specific business processes, improving business efficiency and optimization.

[0658] The processing flow will be explained below.

[0659] Step 1:

[0660] Device:

[0661] When a user starts working on their PC, the client software automatically launches in the background. The client software records operation data in real time, including clicks, keystrokes, application launches and exits, and file operations. This operation data includes the date and time of the operation, as well as details of the application and file involved.

[0662] Step 2:

[0663] Device:

[0664] The recorded operation data is sent to the server at regular intervals. The frequency of data transmission can be adjusted based on network load and security considerations. For example, it can be set to send operation data in batch processing every hour.

[0665] Step 3:

[0666] server:

[0667] The operation data received by the server is stored in a database in real time. The database stores the operation history for each user in chronological order and is used for later analysis. The stored data includes detailed information about the operation and the application in question.

[0668] Step 4:

[0669] server:

[0670] The stored operation data is analyzed using a machine learning algorithm. The algorithm extracts frequently performed operation patterns from the operation data and identifies tasks that are repeated according to specific rules. For example, an operation pattern in which a user enters data into a spreadsheet at the same time every day can be identified.

[0671] Step 5:

[0672] Generated AI (server):

[0673] Based on the identified repetitive tasks, the generative AI suggests the best way to automate them. These suggestions may include generating scripts or using RPA (Robotic Process Automation) tools. The suggestions are customized for each user, showing the most efficient method.

[0674] Step 6:

[0675] server:

[0676] Based on the proposed automation method, specific scripts and configuration files are generated, which contain detailed instructions for the tasks to be automated, and are then prepared for deployment to the user's devices.

[0677] Step 7:

[0678] Device:

[0679] Receive deployed scripts and automation tools from the server, install and configure them according to the configuration files, test the automation tools to ensure they are working properly, and make adjustments as needed.

[0680] Step 8:

[0681] User:

[0682] Execute the automated task and see the results. For example, by automating data entry into a spreadsheet, a task that was previously done manually can now be done instantly by a script. Users can then use the time saved by automation to focus on other creative tasks.

[0683] Step 9:

[0684] server:

[0685] Regularly monitor the use of automation tools and evaluate their effectiveness, checking whether users are becoming more efficient and making additional suggestions or adjustments if further improvements are needed.

[0686] Through the above series of processing steps, users are freed from repetitive routine tasks and can concentrate on creative work.

[0687] Example 1

[0688] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0689] In modern work environments, there are many repetitive tasks that users perform repeatedly. However, these tasks are often performed manually, significantly reducing efficiency. It is also difficult for users to determine which tasks should be automated, and selecting and implementing appropriate automation tools is time-consuming. Therefore, there is a need for a system that records user operations, automatically identifies repetitive tasks, and proposes and implements the optimal automation methods.

[0690] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0691] In this invention, the server includes means for recording user operations in real time, means for periodically transmitting the recorded operation data to the server, means for storing the received data in a database, means for analyzing the stored data using a machine learning algorithm to identify repetitive tasks, means for proposing appropriate automation measures based on the identified repetitive tasks, means for generating scripts or tools that implement the proposed automation measures and installing them on the user's terminal, and means for verifying that the automation tools operate normally on the user's terminal. This frees users from repetitive tasks, allowing them to use their time efficiently and focus on more valuable tasks.

[0692] "User operations" refer to actions such as clicking, keyboard input, starting and closing applications, and file operations that a user performs on a device.

[0693] "Real-time recording means" refers to software or a system that instantly records user operations and saves the data.

[0694] "Operation data" is digital data that includes the content, date and time, details of a user's operation, and information about the application being used.

[0695] The "means for transmitting to the server" refers to a communication protocol or program for transmitting the operation data recorded on the terminal to the server.

[0696] The "means for storing in a database" refers to a system or method for storing the received operation data in a database in a form that allows it to be uniquely identified.

[0697] A "machine learning algorithm" is an algorithm or model that analyzes stored operation data and identifies specific operation patterns.

[0698] A "repetitive task" is an operation or task that is performed in the same sequence and is repeated at regular intervals.

[0699] An "automation tool" is a tool, software, script, or system used to perform repetitive tasks automatically.

[0700] A "script or tool generating means" is a system or software for generating code or configuration files to implement the proposed automation means.

[0701] The "means for introducing the script or tool into the user's terminal" refers to a method or process for placing the generated script or tool on the user's terminal and running it properly.

[0702] "Means to verify proper operation" refers to the process or method for verifying that the implemented scripts and tools operate as intended.

[0703] MODE FOR CARRYING OUT THE INVENTION

[0704] The present invention is a system that automatically identifies repetitive tasks by recording and analyzing user operations, and then proposes and implements appropriate automation measures. This system installs client software on the user's terminal, and analyzes and stores data on a server. Specific embodiments of the system are described below.

[0705] System Overview

[0706] The system records user operations in real time, transmits the operation data to a server, and stores the received data in a database. It uses a machine learning algorithm to analyze operation patterns and identify repetitive tasks. Based on the identified tasks, it proposes automation methods and includes a means for implementing them on the user's device.

[0707] Hardware and software used

[0708] Device:

[0709] Client software is installed on the user's PC, and this software has the function of recording user operations in real time and periodically sending the data to the server.

[0710] server:

[0711] The server stores the received operation data in a database and analyzes it using machine learning algorithms, such as TensorFlow, a machine learning library. Based on the analysis results, it proposes automation methods, generates scripts and tools, and deploys them to the user's device.

[0712] Database:

[0713] The database stores operational data in a uniquely identifiable form, which is then used for later analysis by machine learning algorithms.

[0714] Specific actions

[0715] User operation record:

[0716] The client software on the device records user actions such as clicks, keyboard input, and application launches and exits, generating detailed operational data.

[0717] Data transmission and storage:

[0718] The recorded operation data is sent to a server, for example, every five minutes, and the server stores the received data in a database for later analysis.

[0719] Data analysis and automation methods proposed:

[0720] The server analyzes the stored data using machine learning algorithms to identify repetitive tasks, and then suggests the best automation method (script or tool) based on the identified tasks.

[0721] Generate and deploy scripts and tools:

[0722] The proposed automation methods are embodied as scripts or tools and deployed to the user's terminals, for example, Python scripts or RPA tools.

[0723] Check the operation of the automation tool:

[0724] Users ensure that automated scripts and tools work properly, automating repetitive tasks and improving work efficiency.

[0725] Specific examples

[0726] Example 1: Creating a market analysis report

[0727] Device:

[0728] This tool records the actions of users who periodically collect market data and create reports in Excel. It also records the actions of downloading data from market data sites and entering it into Excel sheets.

[0729] server:

[0730] The recorded operation data is analyzed and "downloading market data" and "entering it into an Excel spreadsheet" are identified as repetitive tasks.

[0731] Generated AI (server):

[0732] We propose a Python script to automate the collection of market data and inputting it into Excel.

[0733] server:

[0734] Generates the proposed Python script and deploys it to the user's device.

[0735] Device:

[0736] When the user runs the script, market data is automatically retrieved and an Excel report is automatically generated.

[0737] User:

[0738] Spend less time creating reports and more time analyzing data and making decisions.

[0739] Prompt Sentence Examples

[0740] "Please suggest a script to record and automate the regular market data collection and Excel report creation."

[0741] Example 2: Managing meeting schedules

[0742] Device:

[0743] This records the actions of users who schedule regular meetings, set up meetings in their Outlook calendar, and invite attendees.

[0744] server:

[0745] By analyzing the operation data, "scheduling a meeting" and "inviting attendees" are identified as repetitive tasks.

[0746] Generated AI (server):

[0747] Suggest a VBA macro or Python script to automate meeting scheduling in Outlook.

[0748] server:

[0749] Generate suggested macros and scripts and prepare them for deployment to users' devices.

[0750] Device:

[0751] Macros and scripts automatically schedule meetings and invite attendees.

[0752] User:

[0753] Free yourself from the manual work of scheduling meetings and use that time to coordinate and execute important projects.

[0754] Prompt Sentence Examples

[0755] "Please suggest a macro or script that records and automates routine meeting setup and attendee invitation actions."

[0756] This frees users from repetitive tasks, allowing them to focus on more creative and strategic work.

[0757] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0758] Step 1:

[0759] Operation Record

[0760] Device:

[0761] Client software installed on the user's PC records the user's actions in real time. Specifically, each time the user clicks, the location information, the details of the action (e.g., button click, form entry), and the application being used are captured. For example, the user entering "sales data" and setting a filter in a spreadsheet is recorded.

[0762] input:

[0763] User actions (clicks, keyboard input, etc.)

[0764] output:

[0765] Operation data (location information, action details, application information, etc.)

[0766] Step 2:

[0767] Data transmission

[0768] Device:

[0769] At regular intervals (for example, every 5 minutes), the recorded operation data is sent to the server. The sent data includes the date and time of the operation, detailed content, and information about the application used.

[0770] input:

[0771] Recorded operation data

[0772] output:

[0773] Operation data sent to the server

[0774] Step 3:

[0775] Data storage

[0776] server:

[0777] The received data is saved in a database. An ID and timestamp are assigned to the database so that operation history can be stored in a uniquely identifiable form. For example, the operation "Enter sales data into a spreadsheet" performed on "2023-01-01 12:00:00" is saved as "Operation History ID: 12345."

[0778] input:

[0779] Operation data sent to the server

[0780] output:

[0781] Operational data stored in a database

[0782] Step 4:

[0783] Data analysis

[0784] server:

[0785] The stored data is analyzed using machine learning algorithms. For example, machine learning libraries such as TensorFlow are used. Through the analysis, frequently performed operations or tasks that are repeated based on certain rules are identified as "repetitive tasks." For example, it may be detected that "entering sales data into a spreadsheet" is performed every day.

[0786] input:

[0787] Operational data stored in a database

[0788] output:

[0789] Repetitive task identification results

[0790] Step 5:

[0791] Proposal of automation methods

[0792] Generated AI (server):

[0793] Based on the analysis results, the system proposes optimal automation methods for identified repetitive tasks. Specifically, it generates scripts or bots (e.g., Python scripts or RPA tools) that mimic the operational procedures and suggests them to the user. For example, it proposes generating a Python script that automatically inputs sales data into a spreadsheet.

[0794] input:

[0795] Repetitive task identification results

[0796] output:

[0797] Proposal of automation methods (scripts and tool specifications)

[0798] Step 6:

[0799] Generating scripts and tools

[0800] server:

[0801] Generate scripts and configuration files to implement the proposed automation measures, for example, generate the proposed Python scripts and package them into a ZIP file.

[0802] input:

[0803] Proposal of automation methods (scripts and tool specifications)

[0804] output:

[0805] Concretized scripts and tools (e.g. generated Python scripts)

[0806] Step 7:

[0807] Deploying scripts and tools

[0808] server:

[0809] Prepare the generated scripts and tools for deployment to the user's device and perform the necessary configuration.

[0810] Device:

[0811] Install the generated scripts and tools on the terminal and register them in the task scheduler to run at 10:00 every day.

[0812] input:

[0813] Embedding scripts and tools (ZIP files, etc.)

[0814] output:

[0815] Scripts and tools deployed on user devices

[0816] Step 8:

[0817] Operation check

[0818] User:

[0819] Ensure automated scripts and tools are working properly and, if necessary, learn how to use them. Well-working scripts automate repetitive tasks and improve work efficiency.

[0820] input:

[0821] Scripts and tools deployed on user devices

[0822] output:

[0823] Confirmed working automation tools

[0824] (Application example 1)

[0825] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0826] The large amount of repetitive manual work at logistics centers consumes a lot of labor and time, resulting in issues such as reduced productivity and increased human error. In particular, efficient automation is required for routine tasks such as picking and inventory taking. Automating these tasks will improve operational efficiency and reduce the burden on workers.

[0827] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0828] In this invention, the server includes means for recording user operations, means for analyzing the recorded operations and identifying repetitive tasks, means for proposing automation measures based on the identified repetitive tasks, means for installing the proposed automation measures on the user's terminal, means for transmitting and storing operation data recorded in logistics operations to a cloud server, means for analyzing the stored data using a machine learning algorithm and identifying repetitive work patterns, and means for generating automation scripts based on the identified work patterns and installing them on the user's mobile device. This makes it possible to efficiently automate repetitive routine tasks performed in logistics centers, reduce the burden on workers, and improve productivity.

[0829] "User" means a person who operates the System or an end user.

[0830] "Means for recording operations" refers to software or devices for capturing and saving user operations in real time.

[0831] A "repetitive task" is work that follows a set pattern, where the same operations are repeated over and over again.

[0832] "Automation tools" are scripts and tools proposed to automate repetitive user tasks.

[0833] "Terminal deployment" refers to the process of installing or making executable the automation means on the user's device.

[0834] A "cloud server" is an external server that stores and analyzes data via the Internet.

[0835] A "machine learning algorithm" is an algorithm that learns patterns from large amounts of data and makes predictions and classifications for new data.

[0836] A "work pattern" is a repetitive action or sequence of operations performed by a user.

[0837] An "automation script" is a program or code created to perform a specific task automatically.

[0838] A "mobile device" is a portable electronic device such as a smartphone or tablet.

[0839] "Storage means" refers to a memory or database for reliably storing recorded data.

[0840] "Proposed means" are functions or processes that show users the automated means.

[0841] "Means for analysis" refers to the systems and processes used to analyze the recorded operational data and extract meaningful information.

[0842] This invention is a system that aims to automate repetitive tasks in logistics centers and maximize operational efficiency. Specifically, it records user operations in real time and analyzes them using machine learning algorithms to identify repetitive tasks and propose and implement automation methods.

[0843] Overall system configuration

[0844] 1. A means of recording user actions

[0845] The client application installed on the device records logistics operations in real time, such as QR code scanners and touchscreen input, including clicks, keyboard input, application launches and exits, and file operations.

[0846] 2. Means of transmitting and storing recorded operating data

[0847] The operation data recorded on the device is periodically sent to a cloud server and stored in AWS or a database (SQLite, PostgreSQL, etc.). This data includes the date and time of the operation and details.

[0848] 3. Data analysis and identification of repetitive tasks

[0849] The server analyzes the stored operation data using machine learning algorithms (e.g., Python, TensorFlow, etc.), thereby identifying frequently performed operations and tasks that are repeated based on certain rules.

[0850] 4. Proposal of automation methods

[0851] The server uses a generative AI model to suggest optimal automation strategies based on identified repetitive tasks, taking into account the frequency and priority of the tasks. For example, a script may be generated for a specific item picking or periodic inventory task.

[0852] 5. Creating and implementing automation tools

[0853] Based on the proposed automation measures, scripts and configuration files are generated and deployed to users' mobile devices (smartphones and tablets). The scripts automate certain repetitive tasks, such as automatically updating new inventory data.

[0854] 6. Automation and operational efficiency

[0855] Using the generated automation tools, users can streamline logistics operations. By automating picking and inventory tasks, the burden on workers is reduced, allowing them to spend more time focusing on important tasks.

[0856] Specific examples

[0857] Example 1: Automated product picking

[0858] At a logistics center, the system records the product picking work performed by workers every day. The data sent to a cloud server is analyzed and repetitive picking patterns are identified. Based on the results, a script to automate the picking work is generated and deployed to smartphones. Workers can use their smartphones to automate picking work, allowing them to work more efficiently.

[0859] Example prompts to input to a generative AI model:

[0860] Generate a script to automate the daily picking tasks performed by distribution center workers. You analyzed the following operational data and identified repetitive picking patterns. Provide an example of a script that would automate this task.

[0861] This system efficiently automates repetitive routine tasks at logistics centers, significantly reducing the burden on workers. By utilizing a combination of generative AI models and machine learning algorithms, it achieves advanced operational efficiency.

[0862] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0863] Step 1:

[0864] When working in the distribution center, users use a client application on their smartphones that uses a QR code scanner or touch screen to record operational data such as picking and inventory in real time.

[0865] Input: Product identification information and work instructions

[0866] Output: Data recording the type and time of operation

[0867] Specific operation: Scan the QR code and save the operation details and timestamp to local storage.

[0868] Step 2:

[0869] The device periodically transmits the recorded data to the cloud server, including detailed information about the operation, the date and time, and information about the working environment.

[0870] Input: Operation data stored in local storage

[0871] Output: Data transferred to the cloud server

[0872] What happens: A scheduled job collects data and sends it over the internet to a cloud server.

[0873] Step 3:

[0874] The server stores the received operational data in a database, where it is formatted and preprocessed for analysis.

[0875] Input: Operation data sent

[0876] Output: Formatted operation data stored in the database

[0877] What happens: The server's receiving module parses the data and imports it into the database in the appropriate format.

[0878] Step 4:

[0879] The server runs machine learning algorithms on the stored operational data to identify repetitive work patterns, using algorithms such as Python and TensorFlow.

[0880] Input: Operational data in the database

[0881] Output: A list of identified recurring tasks

[0882] What it does: Train and evaluate machine learning models to extract important patterns.

[0883] Step 5:

[0884] The server uses a generative AI model to suggest automation measures based on the identified repetitive tasks, and the suggested automation measures are generated in the form of scripts.

[0885] Input: A list of identified recurring tasks

[0886] Output: Automation script

[0887] Specific operation: Provide input data to the generative AI model and obtain the generated script.

[0888] Step 6:

[0889] The proposed automation script is deployed to the device, where the user receives the automation script and prepares it to run on the device.

[0890] Input: Automation script

[0891] Output: Script delivered to the terminal

[0892] Specific operation: The server sends the script to the device, and the application on the device installs or updates it.

[0893] Step 7:

[0894] Users run automation scripts on their devices, which automatically process designated tasks and re-record operation data, improving work efficiency.

[0895] Input: Installed automation script

[0896] Output: The result of the automated task

[0897] What it does: The user launches the script, and the device automatically performs the task and displays the results.

[0898] Through this series of steps, repetitive tasks within the logistics center can be automated, dramatically improving work efficiency.

[0899] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0900] System Overview

[0901] This invention achieves higher business efficiency by combining a system that automatically identifies repetitive tasks by recording and analyzing user operations and proposes and implements appropriate automation measures with an emotion engine that recognizes user emotions. This system involves installing client software and an emotion engine on the user's device and a series of processes that analyze and store operation data and emotion data on a server.

[0902] Program processing

[0903] In the embodiment of the present invention, the following program processing is performed.

[0904] 1. Operation Record and Emotion Recognition

[0905] Device:

[0906] When a user starts working on their PC, the client software and emotion engine automatically start in the background. The client software records operation data such as clicks, keystrokes, application startup / exit, and file operations in real time. At the same time, the emotion engine analyzes the user's facial expressions and tone of voice via the webcam and microphone to identify the user's emotional state.

[0907] 2. Data transmission and storage

[0908] Device:

[0909] The recorded operation data and emotion data are sent to the server at regular intervals. The frequency of data transmission can be adjusted based on network load and security considerations. For example, it can be set to send operation data and emotion data in batch processing every hour.

[0910] server:

[0911] The received operation data and emotion data are stored in a database in real time. The database stores each user's operation history and emotion history in chronological order and is used for later analysis. The stored data includes details of the operation, the target application, and the user's emotional state.

[0912] 3. Data analysis and identification of repetitive tasks

[0913] server:

[0914] The system analyzes the saved operation data using a machine learning algorithm to identify specific operation patterns. Frequently performed operations and tasks that are repeated according to certain rules are identified as repetitive tasks. At the same time, it analyzes emotional data to identify which tasks cause stress or frustration for the user. For example, while identifying an operation pattern in which a user inputs data into a spreadsheet at the same time every day, it also captures emotional data indicating the user is frustrated while performing the task.

[0915] 4. Proposal of automation methods

[0916] Generated AI (server):

[0917] The generative AI then proposes optimal automation methods based on the identified repetitive tasks and emotional data. These proposals include script generation and the use of RPA (Robotic Process Automation) tools. Automation methods are prioritized for tasks that are likely to cause stress to users. For example, a script for automating data entry into a spreadsheet is proposed, and emotional data indicates that automation is urgently needed.

[0918] 5. Creating and implementing automation tools

[0919] server:

[0920] Based on the proposed automation method, specific scripts and configuration files are generated, which contain detailed instructions for the tasks to be automated, and are then prepared for deployment to the user's devices.

[0921] Device:

[0922] Receive deployed scripts and automation tools from the server, install and configure them according to the configuration files, test the automation tools to ensure they are working properly, and make adjustments as needed.

[0923] 6. Implementing automation and improving business efficiency

[0924] User:

[0925] Execute the automated task and observe the effect. For example, by automating data entry into a spreadsheet, a task that was previously done manually can now be done instantly by a script. Also, observe that the emotion engine reduces the user's stress level. Users can use the time saved by automation to pursue other creative tasks.

[0926] Specific examples

[0927] Example 1: Creating a market analysis report and managing stress

[0928] Device:

[0929] The system records the user's actions as they periodically collect market data and create reports in Excel. At the same time, an emotion engine detects whether the user is feeling stressed from their facial expressions while creating the reports.

[0930] server:

[0931] The recorded operation data and emotion data are analyzed to identify whether report creation is occurring regularly and in the same pattern, and to identify that the user is feeling stressed while performing the task.

[0932] Generated AI (server):

[0933] They propose a script to automate the collection of market data and inputting it into Excel, and determine that automation is urgently needed based on sentiment data.

[0934] server:

[0935] Generate the proposed script and deploy it to the user's device.

[0936] Device:

[0937] When the user runs the script, market data is automatically retrieved and an Excel report is automatically created.

[0938] User:

[0939] Time spent on report creation is reduced and can be spent on data analysis and decision-making, reducing stress and improving work efficiency.

[0940] Example 2: Meeting schedule management and emotion monitoring

[0941] Device:

[0942] The system records the user's actions to schedule regular meetings, such as setting up meetings using Outlook Calendar and inviting attendees. The emotion engine also detects frustration from the user's facial expressions during the meeting setup process.

[0943] server:

[0944] Analysis of the operation data identifies scheduling a meeting as a repetitive task that is performed periodically, and emotion data reveals that the task is stressful for the user.

[0945] Generated AI (server):

[0946] Suggest a macro or script to automate meeting scheduling in Outlook, and determine based on sentiment data that automation of this task should be a priority.

[0947] server:

[0948] Generate suggested macros and scripts and prepare them for deployment to users' devices.

[0949] Device:

[0950] Macros and scripts automatically schedule meetings and invite attendees.

[0951] User:

[0952] This frees employees from manually scheduling meetings and allows them to focus on coordinating and implementing important projects. Emotional data helps reduce stress and improve overall work efficiency.

[0953] In this way, the present invention improves the efficiency of users' work, and the emotion engine monitors the user's emotional state and prioritizes the automation of stressful tasks, allowing users to work in a more comfortable environment.

[0954] The processing flow will be explained below.

[0955] Step 1:

[0956] Device:

[0957] When a user starts working on their PC, the client software and emotion engine automatically start in the background. The client software records user operation data in real time, such as clicks, keystrokes, application launches and exits, and file operations. At the same time, the emotion engine analyzes the user's facial expressions and tone of voice via the webcam and microphone to identify the user's emotional state in real time.

[0958] Step 2:

[0959] Device:

[0960] The recorded operation data and emotion data are sent to a server at regular intervals, for example, every 30 minutes, and are encrypted to ensure data integrity.

[0961] Step 3:

[0962] server:

[0963] The received operation data and emotion data are stored in a database. The stored data includes the operation details (clicks, input, file operations, etc.) and the user's emotional state (stress, frustration, satisfaction, etc.) along with the date and time.

[0964] Step 4:

[0965] server:

[0966] The stored data is then analyzed using machine learning algorithms. Operational data is used to extract frequent patterns and identify repetitive tasks, while emotional data is used to understand the emotional state a user is experiencing while performing a particular task. For example, it can identify that a user is experiencing high levels of stress while regularly entering data into a spreadsheet.

[0967] Step 5:

[0968] Generated AI (server):

[0969] Based on the identified repetitive tasks and emotion data, the system suggests automation solutions, including script generation and the use of RPA (Robotic Process Automation) tools. Automation solutions are also prioritized for tasks that users find stressful. For example, a script to automate data entry into a spreadsheet is suggested.

[0970] Step 6:

[0971] server:

[0972] Based on the proposed automation measures, specific scripts and configuration files are generated, which include detailed instructions for executing the automation measures, and the generated scripts and configuration files are prepared for deployment to user devices.

[0973] Step 7:

[0974] Device:

[0975] Receives deployed scripts and automation tools from the server, installs them according to the configuration file, and then tests whether the automation tool works properly and adjusts the configuration if necessary.

[0976] Step 8:

[0977] User:

[0978] Run automated tasks and see the results: for example, an automated script instantly inputs data into a spreadsheet, and the emotion engine reduces stress levels.

[0979] Step 9:

[0980] server:

[0981] Regularly monitor the usage and sentiment data of automation tools to evaluate their effectiveness, confirming that automation is improving work efficiency and reducing user frustration, and make additional suggestions or adjust settings as needed.

[0982] Example 2

[0983] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0984] While conventional task automation systems were able to record and analyze user operations, it was difficult to propose automation methods that took the user's emotional state into account. As a result, it was not possible to appropriately prioritize automation for tasks that users found stressful, resulting in insufficient improvement in work efficiency.

[0985] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0986] In this invention, the server includes means for recording user operations, means for analyzing the recorded operations and identifying repetitive tasks, means for proposing automation measures based on the identified repetitive tasks and emotion data, means for installing the proposed automation measures in the user's terminal, and means for transmitting the recorded operation data and emotion data to the server at regular intervals and storing them in a database. This allows the user to appropriately set automation priorities for tasks that cause stress, thereby improving work efficiency.

[0987] "Means for recording user operations" refers to software or hardware that has the ability to record in real time clicks, keystrokes, application startup / exits, file operations, etc., performed by a user on a PC or other device.

[0988] The "means for identifying repetitive tasks" is a component that analyzes recorded user operation data and runs machine learning algorithms or other analytical techniques to identify frequently performed operations or tasks that are repeated based on certain rules.

[0989] An "emotion engine" is a piece of software or hardware that analyzes a user's facial expressions, tone of voice, etc. to identify the user's emotional state in real time. Specifically, it has the function of detecting the user's emotions using a webcam or microphone.

[0990] "Automation suggestion method" refers to an AI model or rule-based system that suggests optimal automation methods to users based on identified repetitive tasks and sentiment data, including script generation and the use of RPA tools.

[0991] "Means for introducing into the user's terminal" refers to the process for installing and configuring the proposed automation means into the user's terminal and introducing it so that it operates normally.

[0992] The "means for transmitting to a server at regular intervals" refers to a system that has the function of transmitting the recorded operation data and emotion data to a server at specific time intervals in batch processing or in real time.

[0993] The "means for saving to a database" is a component that stores the received operation data and emotion data in a database in a format that can be analyzed later. The data is saved in chronological order and used for later analysis and search.

[0994] This invention achieves higher business efficiency by combining a system that automatically identifies repetitive tasks by recording and analyzing user operations, and proposes and implements appropriate automation measures with an emotion engine that recognizes user emotions. This system involves installing client software and an emotion engine on the user's device, and a series of processes that analyze and store operation data and emotion data on a server.

[0995] A specific embodiment of the system is given below.

[0996] System Overview

[0997] The system consists of client software and an emotion engine installed on the user's device, and a server that analyzes and stores data. The client software records operation data such as clicks, keystrokes, application launches and exits, and file operations. The emotion engine analyzes the user's facial expressions and tone of voice via a webcam and microphone to identify their emotional state.

[0998] Collecting Operational and Emotional Data

[0999] User:

[1000] When a user starts working on their PC, the client software and emotion engine automatically start up in the background, recording all of the user's actions and monitoring the user's emotional state in real time.

[1001] Data transmission and storage

[1002] Device:

[1003] The recorded operation data and emotion data are sent to a server at regular intervals. For example, it is possible to set it to send data in batches every hour. This transmission uses an encrypted communication protocol to ensure the security of the data.

[1004] server:

[1005] The received operation data and emotion data are stored in a database in real time. The operation history and emotion history for each user are stored in chronological order and used for later analysis. For example, details such as what application a specific user used on what date and month, and what tasks they felt stressed during are stored.

[1006] Data analysis and identification of repetitive tasks

[1007] server:

[1008] The saved operation data is analyzed using a machine learning algorithm to identify specific operation patterns. Frequently performed operations and tasks that are repeated according to certain rules are identified as repetitive tasks. At the same time, emotional data is analyzed to identify which tasks the user is experiencing stress or dissatisfaction with. For example, if a user enters data into a spreadsheet at the same time every day, emotional data indicating stress while performing that task can be captured.

[1009] Proposal of automation methods

[1010] Generated AI (server):

[1011] The generative AI proposes optimal automation methods based on the identified repetitive tasks and emotion data. These proposals include generating scripts and using RPA (Robotic Process Automation) tools. Automation methods are prioritized for tasks that are particularly stressful for users. Specifically, if the AI ​​determines that a user is repeatedly entering the same data into Excel and is experiencing high levels of stress while doing so, it will propose a script to automate that task.

[1012] Creating and deploying automation tools

[1013] server:

[1014] According to the proposed automation method, specific scripts and configuration files are generated, including Python scripts and configuration files for RPA (Robotic Process Automation) tools, which are then ready to be deployed to the user's devices.

[1015] Device:

[1016] Receive deployed scripts and automation tools from the server, then install and configure them based on the configuration file. After installation, test the automation tools to ensure they work properly and make any necessary adjustments. For example, verify that the generated scripts automatically input data into a spreadsheet.

[1017] Implementing automation and verifying operational efficiency

[1018] User:

[1019] Execute the automated task and check the effect. For example, by automating data entry into a spreadsheet, the previously manual data entry work can be completed instantly. The emotion engine also confirms a reduction in the user's stress level. This allows the time that would have been spent on manual tasks to be used for other tasks. In particular, work can be carried out with less stress, improving overall work efficiency.

[1020] Specific examples

[1021] Example 1: Creating a market analysis report and managing stress

[1022] Device:

[1023] The system records the user's actions as they periodically collect market data and create reports in Excel, while the emotion engine detects stress from the user's facial expressions while creating the reports.

[1024] server:

[1025] The analysis identifies regular and patterned report creation tasks and the stress experienced by users during those tasks.

[1026] Generated AI (server):

[1027] He proposes a script to automate the collection of market data and inputting it into Excel, and determines that this is an urgent need.

[1028] server:

[1029] Generate the proposed script and deploy it to the user's device.

[1030] Device:

[1031] When you run the script, it will automatically retrieve market data and generate an Excel report.

[1032] User:

[1033] The time spent on creating reports is reduced, and stress is reduced. This allows the time saved to be used for other tasks.

[1034] Example 2: Meeting schedule management and emotion monitoring

[1035] Device:

[1036] The system records users' recurring meeting scheduling actions, including setting up meetings using Outlook Calendar and inviting users to meetings. The emotion engine detects when users experience frustration during the scheduling process.

[1037] server:

[1038] Scheduling regular meetings is identified as a repetitive task, and the task proves to be stressful for the user.

[1039] Generated AI (server):

[1040] Suggest a macro or script to automate meeting scheduling in Outlook and determine that automating this task should be a priority.

[1041] server:

[1042] Generate suggested macros and scripts and deploy them to the user's device.

[1043] Device:

[1044] Automated macros and scripts schedule meetings and automatically invite people.

[1045] User:

[1046] Free yourself from manually scheduling meetings, reducing stress and freeing up time for more important work.

[1047] In this way, the present invention improves the efficiency of users' work and realizes the preferential automation of stressful tasks based on the emotion engine.

[1048] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1049] Step 1:

[1050] User:

[1051] The user starts working on the PC.

[1052] Device:

[1053] When a user logs in to their PC and begins work, the client software and emotion engine automatically start in the background. The client software records the user's clicks, keystrokes, application launches and exits, file operations, and other operational data. At the same time, the emotion engine uses the webcam and microphone to analyze the user's facial expressions and tone of voice, identifying and recording their emotional state in real time.

[1054] Input: User operations, facial expressions, and voice data at the start of the task

[1055] Output: Recorded operation data and emotion data

[1056] Step 2:

[1057] Device:

[1058] The recorded operation data and emotion data are sent to a server at regular intervals using an encrypted communication protocol to ensure data security. For example, the data is sent in batches every hour.

[1059] Input: Recorded operation data and emotion data

[1060] Output: Encrypted data sent to the server

[1061] Step 3:

[1062] server:

[1063] The server receives operation data and emotional data and stores it in a database in real time. Each user's operation history and emotional history are managed in chronological order and used for later analysis. Specific operation patterns and emotional changes can be found from the stored data.

[1064] Input: Operation data and emotion data sent to the server

[1065] Output: Operation history and emotion history stored in a database

[1066] Step 4:

[1067] server:

[1068] The saved operation data is analyzed using machine learning algorithms to identify specific operation patterns and repetitive tasks, while emotional data is analyzed to identify which tasks cause stress to the user. For example, if a user enters data into a spreadsheet at the same time every day, the system can identify a pattern of stress when doing so.

[1069] Input: Operational and emotional data stored in a database

[1070] Output: Identified repetitive tasks and emotional state data

[1071] Step 5:

[1072] Generated AI (server):

[1073] The generative AI proposes optimal automation methods based on the identified repetitive tasks and emotion data. This includes generating scripts and using RPA tools. In particular, it prioritizes automation methods for tasks that are likely to cause stress to the user. For example, if it determines that a user is repeatedly entering the same data into Excel and is experiencing high levels of stress while doing so, it proposes a script to automate that task.

[1074] Input: Identified repetitive tasks and emotional state data

[1075] Output: Proposed automation method (script, RPA tool, etc.)

[1076] Step 6:

[1077] server:

[1078] According to the proposed automation method, specific scripts and configuration files are generated, including Python scripts and configuration files for the RPA tool, which are then ready to be deployed to the user's machine.

[1079] Input: Proposed automation method

[1080] Output: Generated scripts and configuration files

[1081] Step 7:

[1082] Device:

[1083] Receive deployed scripts and automation tools from the server, install and configure them, and test the automation tools to ensure they work properly and adjust them as needed. For example, ensure that a generated script that automatically inputs data into a spreadsheet works properly.

[1084] Input: Scripts and automation tools deployed from the server

[1085] Output: Installed and configured automation tools

[1086] Step 8:

[1087] User:

[1088] Implement automated tasks and see the results. For example, automating data entry into a spreadsheet reduces manual work, and the emotion engine reduces user stress levels. This improves operational efficiency and frees up time spent on manual tasks for other important tasks.

[1089] Input: Installed automation tools

[1090] Output: The automated tasks performed and their operational efficiency effects

[1091] (Application example 2)

[1092] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1093] While conventional business efficiency systems can record user operations and automate repetitive tasks, they do not take into account the user's emotions or stress levels, which means that tasks that cause stress to the user are not automated, making it difficult to achieve effective efficiency. Furthermore, in brick-and-mortar stores, many of the tasks that employees perform daily are repetitive and their emotions fluctuate frequently, so efficiency measures that take stress levels into account are required. This leads to a decrease in employee satisfaction and work efficiency, which has a negative impact on the overall performance of the store.

[1094] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1095] In this invention, the server includes a means for recording user operations, a means for analyzing the recorded operations to identify repetitive tasks, a means for identifying the user's emotions, and a means for analyzing the emotional data and adjusting automation priorities to reduce stress. This enables automation that takes into account not only the user's operation data but also their emotional data, enabling the prioritized automation of tasks that are particularly stressful. As a result, employees are freed from stressful tasks and can focus on more creative and satisfying work, improving the operational efficiency and performance of the entire physical store.

[1096] "Means for recording user operations" refers to a function that collects and saves data such as user clicks, keystrokes, application launches and exits, and file operations in real time.

[1097] "Means for analyzing recorded operations and identifying repetitive tasks" refers to a function that analyzes collected operation data using machine learning algorithms, etc., to detect specific operation patterns that are frequently repeated.

[1098] "Means for suggesting automation measures based on identified repetitive tasks" refers to a function that recommends the use of script generation or automation tools based on data on repetitive tasks.

[1099] "Means for introducing the proposed automation measures into the user's device" refers to the ability to install the proposed automation tools or scripts on the user's device and execute the settings.

[1100] The "means for identifying user emotions" is a function that analyzes the user's facial expressions and tone of voice via a webcam or microphone to identify their emotional state.

[1101] "Means for analyzing emotional data and adjusting automation priorities to reduce stress" refers to a function that identifies tasks that cause stress to users based on emotional data, and then suggests and executes the automation of those tasks as a priority.

[1102] The "means for storing the recorded operation data" is a function for periodically sending the collected operation data to a server or database and storing it.

[1103] "Means for learning operation patterns based on stored data" refers to a function that uses stored data to utilize machine learning algorithms, etc., to identify and learn newly emerging patterns and existing patterns.

[1104] "Means for storing emotional data" refers to a function that periodically sends collected emotional data to a server or database and stores it.

[1105] The "means for analyzing stored emotion data and identifying stressful tasks" is a function for analyzing stored emotion data and identifying tasks that cause stress to the user.

[1106] The "means for evaluating the execution frequency of identified tasks and determining automation priorities based on the frequency" is a function that analyzes task execution frequency data and prioritizes automation of tasks that occur frequently.

[1107] The "means for readjusting automation priorities based on stress level" is a function that reviews the initial automation priorities based on the user's stress level data, thereby achieving more effective stress reduction.

[1108] This invention is a system for improving the work efficiency and reducing stress of employees in brick-and-mortar stores. Specific embodiments for realizing this system will be described below.

[1109] System configuration

[1110] The system includes the following hardware and software:

[1111] Smartphones or smart glasses: Equipped with a camera and microphone, they collect user interaction and emotion data.

[1112] EmotionRecognizer class: Analyzes user emotions using an open-source emotion recognition library.

[1113] TaskAutomator class: A custom library for automating tasks.

[1114] Server: Stores operation data and emotion data and performs data analysis using machine learning algorithms.

[1115] Program processing

[1116] The server first launches an application on a smartphone or smart glasses to collect user operation data in real time. The application records the employee's clicks, keystrokes, system startup and shutdown, etc. It also uses the built-in camera and microphone to collect emotional data from facial expressions and tone of voice.

[1117] The collected data is sent to a server at regular intervals, where it is stored in a database of both user operation and emotion data. The server then uses machine learning algorithms to identify repetitive tasks and analyzes the emotion data to identify tasks that cause stress to the user.

[1118] For high-stress, frequently repetitive tasks, the system uses a generative AI model to suggest optimal automation methods, which are then embodied as scripts or automation tools and deployed from a server to users' devices.

[1119] The system installs the proposed automation tool on the user's device and optimizes it based on the configuration file. During this process, the system checks the operation of the automation tool and makes adjustments as necessary.

[1120] Employees only need to perform automated tasks, significantly reducing the amount of manual work they need to do, which increases work efficiency and reduces stress levels.

[1121] Specific examples

[1122] For example, suppose there is an employee who handles accounting at a cash register in a physical store. In this case, the system operates as follows:

[1123] 1. Operation Record and Emotion Recognition

[1124] Record employees scanning items into the cash register system.

[1125] Using the camera in smart glasses or the microphone in a smartphone, it analyzes employees' facial expressions and voice to identify their emotional state.

[1126] 2. Data transmission and analysis

[1127] The collected operation data and emotion data are sent to a server.

[1128] The server stores and analyzes this data.

[1129] 3. Identifying repetitive tasks and suggesting automation

[1130] Identify that cash register operations are repetitive tasks.

[1131] Emotional data can be used to identify stress experienced while operating a cash register.

[1132] As an automation suggestion, we generate scripts to automate product scanning and accounting processes.

[1133] 4. Implementing automation tools

[1134] The scripts proposed by the server are deployed to the user's device and configured.

[1135] 5. Execute automation

[1136] Automated scripts handle cash register operations, leaving employees to simply verify the transaction.

[1137] In this way, the burden on employees can be reduced and work efficiency can be improved.

[1138] Prompt Sentence Examples

[1139] A sample prompt is as follows:

[1140] "Record cashier activity, analyze sentiment data, and suggest tasks that can be automated, such as scanning items or printing receipts."

[1141] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1142] Step 1:

[1143] When a user begins work on a device, an application installed on the smartphone or smart glasses automatically starts in the background. The application records user operation data in real time, including clicks, keystrokes, system startup / shutdown, and file operations. At the same time, it uses the built-in camera and microphone to collect emotional data from facial expressions and tone of voice. As a result, the input data becomes operation data and emotional data. The output data is temporarily stored on the device as separate datasets.

[1144] Step 2:

[1145] The device transmits the recorded operation data and emotion data to the server at regular intervals. The transmission frequency is adjusted based on network load and security considerations, for example, every 30 minutes. The input to this step is the operation data and emotion data recorded in step 1, and the output is the data transmitted to the server.

[1146] Step 3:

[1147] The operation data and emotion data received by the server are immediately stored in a database. The stored data includes a timestamp, operation details, system information of the operation target, and the user's emotional state. The input of this step is the operation data and emotion data sent from the terminal, and the output is storage in the database.

[1148] Step 4:

[1149] The server uses a machine learning algorithm to analyze the stored data and identify specific operation patterns. Repetitive operations and regularly repeated tasks are identified as repetitive tasks. At the same time, it analyzes emotional data to identify which tasks employees experience stress during. The input for this step is the operation data and emotional data stored in the database, and the output is the identified repetitive tasks and stressful tasks.

[1150] Step 5:

[1151] The server proposes automation methods based on the repetitive tasks and emotion data identified using the generative AI model. The generative AI model provides optimal methods for automating tasks, such as script generation and automation tool configuration. It also adjusts automation priorities according to the user's stress level. The inputs for this step are the repetitive tasks and stress data identified in step 4, and the output is specific automation method proposals.

[1152] Step 6:

[1153] The server instantiates the proposed automation measures and generates scripts and configuration files. The generated data is ready to be deployed to the user's device. The input to this step is the automation measures proposed in step 5, and the output is the generated scripts and configuration files.

[1154] Step 7:

[1155] The terminal receives the deployed scripts and automation tools from the server, installs and configures them on the terminal, and performs the necessary tests to verify that the tools work properly. The input to this step is the scripts and configuration files sent from the server, and the output is the installed automation tools.

[1156] Step 8:

[1157] The user executes the automated task and checks its effectiveness. For example, by automating cash register operations, employees are only required to perform the checks. This reduces work time and lowers stress levels. The input for this step is the installed automation tool, and the output is the results of the automation and the streamlined business data.

[1158] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1159] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1160] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1161] [Third embodiment]

[1162] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1163] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1164] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1165] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1166] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1167] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1168] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1169] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1170] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1171] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1172] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1173] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1174] System Overview

[1175] This invention is a system that automatically identifies repetitive tasks by recording and analyzing user operations, and proposes and implements appropriate automation measures. This system involves a series of processes: installing client software on the user's device, and analyzing and saving operation data on a server.

[1176] Program processing

[1177] In the embodiment of the present invention, the following program processing is performed.

[1178] 1. Operation Record

[1179] Device:

[1180] Client software installed on the user's PC records user actions in real time, including clicks, keyboard input, application startup and shutdown, file operations, etc. For example, a user entering data into a spreadsheet is recorded.

[1181] 2. Data transmission and storage

[1182] Device:

[1183] The recorded operation data is periodically sent to a server, including the date, time, and details of the operation.

[1184] server:

[1185] The received data is stored in a database, allowing all operation histories to be managed together and used for later analysis.

[1186] 3. Data analysis and identification of repetitive tasks

[1187] server:

[1188] The stored operational data is analyzed using machine learning algorithms to identify specific operational patterns. Frequently performed operations or tasks that are repeated based on certain rules are identified as repetitive tasks. For example, entering data into a spreadsheet in the same format every day can be identified.

[1189] 4. Proposal of automation methods

[1190] Generated AI (server):

[1191] Based on the identified repetitive tasks, it recommends the best means of automation, which can include generating scripts or using RPA tools. For example, to automate data entry into a spreadsheet, it suggests a data entry script that follows a specific format.

[1192] 5. Creating and implementing automation tools

[1193] server:

[1194] Scripts and configuration files are generated to implement the proposed automation methods, and the settings are configured to be deployed to the user's device.

[1195] Device:

[1196] Install the generated scripts and tools and execute the automation according to your settings. When a user opens a spreadsheet, the configured scripts will automatically populate the data.

[1197] 6. Implementing automation and improving business efficiency

[1198] User:

[1199] Ensure your automation tools are working properly and, if necessary, learn how to use them, which will free up time to focus on creative and strategic tasks rather than manual, repetitive tasks.

[1200] Specific examples

[1201] Example 1: Creating a market analysis report

[1202] Device:

[1203] The user's actions of periodically collecting market data and creating reports in Excel are recorded.

[1204] server:

[1205] Analyze recorded operational data to identify regular and consistent patterns of reporting.

[1206] Generated AI (server):

[1207] Propose a script to automate the collection of market data and inputting it into Excel.

[1208] server:

[1209] Generate the proposed script and deploy it to the user's device.

[1210] Device:

[1211] When the user runs the script, market data is automatically retrieved and an Excel report is automatically created.

[1212] User:

[1213] Time spent on report generation is reduced and can be used for data analysis and decision making.

[1214] Example 2: Managing meeting schedules

[1215] Device:

[1216] It records actions taken by users when they schedule regular meetings, such as setting up a meeting using Outlook Calendar and inviting attendees.

[1217] server:

[1218] As a result of analyzing the operational data, scheduling meetings is identified as a recurring task that is performed periodically.

[1219] Generated AI (server):

[1220] Suggest a macro or script to automate meeting scheduling in Outlook.

[1221] server:

[1222] Generate suggested macros and scripts and prepare them for deployment to users' devices.

[1223] Device:

[1224] Macros and scripts automatically schedule meetings and invite attendees.

[1225] User:

[1226] Free yourself from manually scheduling meetings and use that time to coordinate and execute important projects.

[1227] This frees users from repetitive routine tasks, allowing them to focus on more advanced tasks. The system generates AI specialized for specific business processes, improving business efficiency and optimization.

[1228] The processing flow will be explained below.

[1229] Step 1:

[1230] Device:

[1231] When a user starts working on their PC, the client software automatically launches in the background. The client software records operation data in real time, including clicks, keystrokes, application launches and exits, and file operations. This operation data includes the date and time of the operation, as well as details of the application and file involved.

[1232] Step 2:

[1233] Device:

[1234] The recorded operation data is sent to the server at regular intervals. The frequency of data transmission can be adjusted based on network load and security considerations. For example, it can be set to send operation data in batch processing every hour.

[1235] Step 3:

[1236] server:

[1237] The operation data received by the server is stored in a database in real time. The database stores the operation history for each user in chronological order and is used for later analysis. The stored data includes detailed information about the operation and the application in question.

[1238] Step 4:

[1239] server:

[1240] The stored operation data is analyzed using a machine learning algorithm. The algorithm extracts frequently performed operation patterns from the operation data and identifies tasks that are repeated according to specific rules. For example, an operation pattern in which a user enters data into a spreadsheet at the same time every day can be identified.

[1241] Step 5:

[1242] Generated AI (server):

[1243] Based on the identified repetitive tasks, the generative AI suggests the best way to automate them. These suggestions may include generating scripts or using RPA (Robotic Process Automation) tools. The suggestions are customized for each user, showing the most efficient method.

[1244] Step 6:

[1245] server:

[1246] Based on the proposed automation method, specific scripts and configuration files are generated, which contain detailed instructions for the tasks to be automated, and are then prepared for deployment to the user's devices.

[1247] Step 7:

[1248] Device:

[1249] Receive deployed scripts and automation tools from the server, install and configure them according to the configuration files, test the automation tools to ensure they are working properly, and make adjustments as needed.

[1250] Step 8:

[1251] User:

[1252] Execute the automated task and see the results. For example, by automating data entry into a spreadsheet, a task that was previously done manually can now be done instantly by a script. Users can then use the time saved by automation to focus on other creative tasks.

[1253] Step 9:

[1254] server:

[1255] Regularly monitor the use of automation tools and evaluate their effectiveness, checking whether users are becoming more efficient and making additional suggestions or adjustments if further improvements are needed.

[1256] Through the above series of processing steps, users are freed from repetitive routine tasks and can concentrate on creative work.

[1257] Example 1

[1258] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1259] In modern work environments, there are many repetitive tasks that users perform repeatedly. However, these tasks are often performed manually, significantly reducing efficiency. It is also difficult for users to determine which tasks should be automated, and selecting and implementing appropriate automation tools is time-consuming. Therefore, there is a need for a system that records user operations, automatically identifies repetitive tasks, and proposes and implements the optimal automation methods.

[1260] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1261] In this invention, the server includes means for recording user operations in real time, means for periodically transmitting the recorded operation data to the server, means for storing the received data in a database, means for analyzing the stored data using a machine learning algorithm to identify repetitive tasks, means for proposing appropriate automation measures based on the identified repetitive tasks, means for generating scripts or tools that implement the proposed automation measures and installing them on the user's terminal, and means for verifying that the automation tools operate normally on the user's terminal. This frees users from repetitive tasks, allowing them to use their time efficiently and focus on more valuable tasks.

[1262] "User operations" refer to actions such as clicking, keyboard input, starting and closing applications, and file operations that a user performs on a device.

[1263] "Real-time recording means" refers to software or a system that instantly records user operations and saves the data.

[1264] "Operation data" is digital data that includes the content, date and time, details of a user's operation, and information about the application being used.

[1265] The "means for transmitting to the server" refers to a communication protocol or program for transmitting the operation data recorded on the terminal to the server.

[1266] The "means for storing in a database" refers to a system or method for storing the received operation data in a database in a form that allows it to be uniquely identified.

[1267] A "machine learning algorithm" is an algorithm or model that analyzes stored operation data and identifies specific operation patterns.

[1268] A "repetitive task" is an operation or task that is performed in the same sequence and is repeated at regular intervals.

[1269] An "automation tool" is a tool, software, script, or system used to perform repetitive tasks automatically.

[1270] A "script or tool generating means" is a system or software for generating code or configuration files to implement the proposed automation means.

[1271] The "means for introducing the script or tool into the user's terminal" refers to a method or process for placing the generated script or tool on the user's terminal and running it properly.

[1272] "Means to verify proper operation" refers to the process or method for verifying that the implemented scripts and tools operate as intended.

[1273] MODE FOR CARRYING OUT THE INVENTION

[1274] The present invention is a system that automatically identifies repetitive tasks by recording and analyzing user operations, and then proposes and implements appropriate automation measures. This system installs client software on the user's terminal, and analyzes and stores data on a server. Specific embodiments of the system are described below.

[1275] System Overview

[1276] The system records user operations in real time, transmits the operation data to a server, and stores the received data in a database. It uses a machine learning algorithm to analyze operation patterns and identify repetitive tasks. Based on the identified tasks, it proposes automation methods and includes a means for implementing them on the user's device.

[1277] Hardware and software used

[1278] Device:

[1279] Client software is installed on the user's PC, and this software has the function of recording user operations in real time and periodically sending the data to the server.

[1280] server:

[1281] The server stores the received operation data in a database and analyzes it using machine learning algorithms, such as TensorFlow, a machine learning library. Based on the analysis results, it proposes automation methods, generates scripts and tools, and deploys them to the user's device.

[1282] Database:

[1283] The database stores operational data in a uniquely identifiable form, which is then used for later analysis by machine learning algorithms.

[1284] Specific actions

[1285] User operation record:

[1286] The client software on the device records user actions such as clicks, keyboard input, and application launches and exits, generating detailed operational data.

[1287] Data transmission and storage:

[1288] The recorded operation data is sent to a server, for example, every five minutes, and the server stores the received data in a database for later analysis.

[1289] Data analysis and automation methods proposed:

[1290] The server analyzes the stored data using machine learning algorithms to identify repetitive tasks, and then suggests the best automation method (script or tool) based on the identified tasks.

[1291] Generate and deploy scripts and tools:

[1292] The proposed automation methods are embodied as scripts or tools and deployed to the user's terminals, for example, Python scripts or RPA tools.

[1293] Check the operation of the automation tool:

[1294] Users ensure that automated scripts and tools work properly, automating repetitive tasks and improving work efficiency.

[1295] Specific examples

[1296] Example 1: Creating a market analysis report

[1297] Device:

[1298] This tool records the actions of users who periodically collect market data and create reports in Excel. It also records the actions of downloading data from market data sites and entering it into Excel sheets.

[1299] server:

[1300] The recorded operation data is analyzed and "downloading market data" and "entering it into an Excel spreadsheet" are identified as repetitive tasks.

[1301] Generated AI (server):

[1302] We propose a Python script to automate the collection of market data and inputting it into Excel.

[1303] server:

[1304] Generates the proposed Python script and deploys it to the user's device.

[1305] Device:

[1306] When the user runs the script, market data is automatically retrieved and an Excel report is automatically generated.

[1307] User:

[1308] Spend less time creating reports and more time analyzing data and making decisions.

[1309] Prompt Sentence Examples

[1310] "Please suggest a script to record and automate the regular market data collection and Excel report creation."

[1311] Example 2: Managing meeting schedules

[1312] Device:

[1313] This records the actions of users who schedule regular meetings, set up meetings in their Outlook calendar, and invite attendees.

[1314] server:

[1315] By analyzing the operation data, "scheduling a meeting" and "inviting attendees" are identified as repetitive tasks.

[1316] Generated AI (server):

[1317] Suggest a VBA macro or Python script to automate meeting scheduling in Outlook.

[1318] server:

[1319] Generate suggested macros and scripts and prepare them for deployment to users' devices.

[1320] Device:

[1321] Macros and scripts automatically schedule meetings and invite attendees.

[1322] User:

[1323] Free yourself from the manual work of scheduling meetings and use that time to coordinate and execute important projects.

[1324] Prompt Sentence Examples

[1325] "Please suggest a macro or script that records and automates routine meeting setup and attendee invitation actions."

[1326] This frees users from repetitive tasks, allowing them to focus on more creative and strategic work.

[1327] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1328] Step 1:

[1329] Operation Record

[1330] Device:

[1331] Client software installed on the user's PC records the user's actions in real time. Specifically, each time the user clicks, the location information, the details of the action (e.g., button click, form entry), and the application being used are captured. For example, the user entering "sales data" and setting a filter in a spreadsheet is recorded.

[1332] input:

[1333] User actions (clicks, keyboard input, etc.)

[1334] output:

[1335] Operation data (location information, action details, application information, etc.)

[1336] Step 2:

[1337] Data transmission

[1338] Device:

[1339] At regular intervals (for example, every 5 minutes), the recorded operation data is sent to the server. The sent data includes the date and time of the operation, detailed content, and information about the application used.

[1340] input:

[1341] Recorded operation data

[1342] output:

[1343] Operation data sent to the server

[1344] Step 3:

[1345] Data storage

[1346] server:

[1347] The received data is saved in a database. An ID and timestamp are assigned to the database so that operation history can be stored in a uniquely identifiable form. For example, the operation "Enter sales data into a spreadsheet" performed on "2023-01-01 12:00:00" is saved as "Operation History ID: 12345."

[1348] input:

[1349] Operation data sent to the server

[1350] output:

[1351] Operational data stored in a database

[1352] Step 4:

[1353] Data analysis

[1354] server:

[1355] The stored data is analyzed using machine learning algorithms. For example, machine learning libraries such as TensorFlow are used. Through the analysis, frequently performed operations or tasks that are repeated based on certain rules are identified as "repetitive tasks." For example, it may be detected that "entering sales data into a spreadsheet" is performed every day.

[1356] input:

[1357] Operational data stored in a database

[1358] output:

[1359] Repetitive task identification results

[1360] Step 5:

[1361] Proposal of automation methods

[1362] Generated AI (server):

[1363] Based on the analysis results, the system proposes optimal automation methods for identified repetitive tasks. Specifically, it generates scripts or bots (e.g., Python scripts or RPA tools) that mimic the operational procedures and suggests them to the user. For example, it proposes generating a Python script that automatically inputs sales data into a spreadsheet.

[1364] input:

[1365] Repetitive task identification results

[1366] output:

[1367] Proposal of automation methods (scripts and tool specifications)

[1368] Step 6:

[1369] Generating scripts and tools

[1370] server:

[1371] Generate scripts and configuration files to implement the proposed automation measures, for example, generate the proposed Python scripts and package them into a ZIP file.

[1372] input:

[1373] Proposal of automation methods (scripts and tool specifications)

[1374] output:

[1375] Concretized scripts and tools (e.g. generated Python scripts)

[1376] Step 7:

[1377] Deploying scripts and tools

[1378] server:

[1379] Prepare the generated scripts and tools for deployment to the user's device and perform the necessary configuration.

[1380] Device:

[1381] Install the generated scripts and tools on the terminal and register them in the task scheduler to run at 10:00 every day.

[1382] input:

[1383] Embedding scripts and tools (ZIP files, etc.)

[1384] output:

[1385] Scripts and tools deployed on user devices

[1386] Step 8:

[1387] Operation check

[1388] User:

[1389] Ensure automated scripts and tools are working properly and, if necessary, learn how to use them. Well-working scripts automate repetitive tasks and improve work efficiency.

[1390] input:

[1391] Scripts and tools deployed on user devices

[1392] output:

[1393] Confirmed working automation tools

[1394] (Application example 1)

[1395] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1396] The large amount of repetitive manual work at logistics centers consumes a lot of labor and time, resulting in issues such as reduced productivity and increased human error. In particular, efficient automation is required for routine tasks such as picking and inventory taking. Automating these tasks will improve operational efficiency and reduce the burden on workers.

[1397] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1398] In this invention, the server includes means for recording user operations, means for analyzing the recorded operations and identifying repetitive tasks, means for proposing automation measures based on the identified repetitive tasks, means for installing the proposed automation measures on the user's terminal, means for transmitting and storing operation data recorded in logistics operations to a cloud server, means for analyzing the stored data using a machine learning algorithm and identifying repetitive work patterns, and means for generating automation scripts based on the identified work patterns and installing them on the user's mobile device. This makes it possible to efficiently automate repetitive routine tasks performed in logistics centers, reduce the burden on workers, and improve productivity.

[1399] "User" means a person who operates the System or an end user.

[1400] "Means for recording operations" refers to software or devices for capturing and saving user operations in real time.

[1401] A "repetitive task" is work that follows a set pattern, where the same operations are repeated over and over again.

[1402] "Automation tools" are scripts and tools proposed to automate repetitive user tasks.

[1403] "Terminal deployment" refers to the process of installing or making executable the automation means on the user's device.

[1404] A "cloud server" is an external server that stores and analyzes data via the Internet.

[1405] A "machine learning algorithm" is an algorithm that learns patterns from large amounts of data and makes predictions and classifications for new data.

[1406] A "work pattern" is a repetitive action or sequence of operations performed by a user.

[1407] An "automation script" is a program or code created to perform a specific task automatically.

[1408] A "mobile device" is a portable electronic device such as a smartphone or tablet.

[1409] "Storage means" refers to a memory or database for reliably storing recorded data.

[1410] "Proposed means" are functions or processes that show users the automated means.

[1411] "Means for analysis" refers to the systems and processes used to analyze the recorded operational data and extract meaningful information.

[1412] This invention is a system that aims to automate repetitive tasks in logistics centers and maximize operational efficiency. Specifically, it records user operations in real time and analyzes them using machine learning algorithms to identify repetitive tasks and propose and implement automation methods.

[1413] Overall system configuration

[1414] 1. A means of recording user actions

[1415] The client application installed on the device records logistics operations in real time, such as QR code scanners and touchscreen input, including clicks, keyboard input, application launches and exits, and file operations.

[1416] 2. Means of transmitting and storing recorded operating data

[1417] The operation data recorded on the device is periodically sent to a cloud server and stored in AWS or a database (SQLite, PostgreSQL, etc.). This data includes the date and time of the operation and details.

[1418] 3. Data analysis and identification of repetitive tasks

[1419] The server analyzes the stored operation data using machine learning algorithms (e.g., Python, TensorFlow, etc.), thereby identifying frequently performed operations and tasks that are repeated based on certain rules.

[1420] 4. Proposal of automation methods

[1421] The server uses a generative AI model to suggest optimal automation strategies based on identified repetitive tasks, taking into account the frequency and priority of the tasks. For example, a script may be generated for a specific item picking or periodic inventory task.

[1422] 5. Creating and implementing automation tools

[1423] Based on the proposed automation measures, scripts and configuration files are generated and deployed to users' mobile devices (smartphones and tablets). The scripts automate certain repetitive tasks, such as automatically updating new inventory data.

[1424] 6. Automation and operational efficiency

[1425] Using the generated automation tools, users can streamline logistics operations. By automating picking and inventory tasks, the burden on workers is reduced, allowing them to spend more time focusing on important tasks.

[1426] Specific examples

[1427] Example 1: Automated product picking

[1428] At a logistics center, the system records the product picking work performed by workers every day. The data sent to a cloud server is analyzed and repetitive picking patterns are identified. Based on the results, a script to automate the picking work is generated and deployed to smartphones. Workers can use their smartphones to automate picking work, allowing them to work more efficiently.

[1429] Example prompts to input to a generative AI model:

[1430] Generate a script to automate the daily picking tasks performed by distribution center workers. You analyzed the following operational data and identified repetitive picking patterns. Provide an example of a script that would automate this task.

[1431] This system efficiently automates repetitive routine tasks at logistics centers, significantly reducing the burden on workers. By utilizing a combination of generative AI models and machine learning algorithms, it achieves advanced operational efficiency.

[1432] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1433] Step 1:

[1434] When working in the distribution center, users use a client application on their smartphones that uses a QR code scanner or touch screen to record operational data such as picking and inventory in real time.

[1435] Input: Product identification information and work instructions

[1436] Output: Data recording the type and time of operation

[1437] Specific operation: Scan the QR code and save the operation details and timestamp to local storage.

[1438] Step 2:

[1439] The device periodically transmits the recorded data to the cloud server, including detailed information about the operation, the date and time, and information about the working environment.

[1440] Input: Operation data stored in local storage

[1441] Output: Data transferred to the cloud server

[1442] What happens: A scheduled job collects data and sends it over the internet to a cloud server.

[1443] Step 3:

[1444] The server stores the received operational data in a database, where it is formatted and preprocessed for analysis.

[1445] Input: Operation data sent

[1446] Output: Formatted operation data stored in the database

[1447] What happens: The server's receiving module parses the data and imports it into the database in the appropriate format.

[1448] Step 4:

[1449] The server runs machine learning algorithms on the stored operational data to identify repetitive work patterns, using algorithms such as Python and TensorFlow.

[1450] Input: Operational data in the database

[1451] Output: A list of identified recurring tasks

[1452] What it does: Train and evaluate machine learning models to extract important patterns.

[1453] Step 5:

[1454] The server uses a generative AI model to suggest automation measures based on the identified repetitive tasks, and the suggested automation measures are generated in the form of scripts.

[1455] Input: A list of identified recurring tasks

[1456] Output: Automation script

[1457] Specific operation: Provide input data to the generative AI model and obtain the generated script.

[1458] Step 6:

[1459] The proposed automation script is deployed to the device, where the user receives the automation script and prepares it to run on the device.

[1460] Input: Automation script

[1461] Output: Script delivered to the terminal

[1462] Specific operation: The server sends the script to the device, and the application on the device installs or updates it.

[1463] Step 7:

[1464] Users run automation scripts on their devices, which automatically process designated tasks and re-record operation data, improving work efficiency.

[1465] Input: Installed automation script

[1466] Output: The result of the automated task

[1467] What it does: The user launches the script, and the device automatically performs the task and displays the results.

[1468] Through this series of steps, repetitive tasks within the logistics center can be automated, dramatically improving work efficiency.

[1469] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1470] System Overview

[1471] This invention achieves higher business efficiency by combining a system that automatically identifies repetitive tasks by recording and analyzing user operations and proposes and implements appropriate automation measures with an emotion engine that recognizes user emotions. This system involves installing client software and an emotion engine on the user's device and a series of processes that analyze and store operation data and emotion data on a server.

[1472] Program processing

[1473] In the embodiment of the present invention, the following program processing is performed.

[1474] 1. Operation Record and Emotion Recognition

[1475] Device:

[1476] When a user starts working on their PC, the client software and emotion engine automatically start in the background. The client software records operation data such as clicks, keystrokes, application startup / exit, and file operations in real time. At the same time, the emotion engine analyzes the user's facial expressions and tone of voice via the webcam and microphone to identify the user's emotional state.

[1477] 2. Data transmission and storage

[1478] Device:

[1479] The recorded operation data and emotion data are sent to the server at regular intervals. The frequency of data transmission can be adjusted based on network load and security considerations. For example, it can be set to send operation data and emotion data in batch processing every hour.

[1480] server:

[1481] The received operation data and emotion data are stored in a database in real time. The database stores each user's operation history and emotion history in chronological order and is used for later analysis. The stored data includes details of the operation, the target application, and the user's emotional state.

[1482] 3. Data analysis and identification of repetitive tasks

[1483] server:

[1484] The system analyzes the saved operation data using a machine learning algorithm to identify specific operation patterns. Frequently performed operations and tasks that are repeated according to certain rules are identified as repetitive tasks. At the same time, it analyzes emotional data to identify which tasks cause stress or frustration for the user. For example, while identifying an operation pattern in which a user inputs data into a spreadsheet at the same time every day, it also captures emotional data indicating the user is frustrated while performing the task.

[1485] 4. Proposal of automation methods

[1486] Generated AI (server):

[1487] The generative AI then proposes optimal automation methods based on the identified repetitive tasks and emotional data. These proposals include script generation and the use of RPA (Robotic Process Automation) tools. Automation methods are prioritized for tasks that are likely to cause stress to users. For example, a script for automating data entry into a spreadsheet is proposed, and emotional data indicates that automation is urgently needed.

[1488] 5. Creating and implementing automation tools

[1489] server:

[1490] Based on the proposed automation method, specific scripts and configuration files are generated, which contain detailed instructions for the tasks to be automated, and are then prepared for deployment to the user's devices.

[1491] Device:

[1492] Receive deployed scripts and automation tools from the server, install and configure them according to the configuration files, test the automation tools to ensure they are working properly, and make adjustments as needed.

[1493] 6. Implementing automation and improving business efficiency

[1494] User:

[1495] Execute the automated task and observe the effect. For example, by automating data entry into a spreadsheet, a task that was previously done manually can now be done instantly by a script. Also, observe that the emotion engine reduces the user's stress level. Users can use the time saved by automation to pursue other creative tasks.

[1496] Specific examples

[1497] Example 1: Creating a market analysis report and managing stress

[1498] Device:

[1499] The system records the user's actions as they periodically collect market data and create reports in Excel. At the same time, an emotion engine detects whether the user is feeling stressed from their facial expressions while creating the reports.

[1500] server:

[1501] The recorded operation data and emotion data are analyzed to identify whether report creation is occurring regularly and in the same pattern, and to identify that the user is feeling stressed while performing the task.

[1502] Generated AI (server):

[1503] They propose a script to automate the collection of market data and inputting it into Excel, and determine that automation is urgently needed based on sentiment data.

[1504] server:

[1505] Generate the proposed script and deploy it to the user's device.

[1506] Device:

[1507] When the user runs the script, market data is automatically retrieved and an Excel report is automatically created.

[1508] User:

[1509] Time spent on report creation is reduced and can be spent on data analysis and decision-making, reducing stress and improving work efficiency.

[1510] Example 2: Meeting schedule management and emotion monitoring

[1511] Device:

[1512] The system records the user's actions to schedule regular meetings, such as setting up meetings using Outlook Calendar and inviting attendees. The emotion engine also detects frustration from the user's facial expressions during the meeting setup process.

[1513] server:

[1514] Analysis of the operation data identifies scheduling a meeting as a repetitive task that is performed periodically, and emotion data reveals that the task is stressful for the user.

[1515] Generated AI (server):

[1516] Suggest a macro or script to automate meeting scheduling in Outlook, and determine based on sentiment data that automation of this task should be a priority.

[1517] server:

[1518] Generate suggested macros and scripts and prepare them for deployment to users' devices.

[1519] Device:

[1520] Macros and scripts automatically schedule meetings and invite attendees.

[1521] User:

[1522] This frees employees from manually scheduling meetings and allows them to focus on coordinating and implementing important projects. Emotional data helps reduce stress and improve overall work efficiency.

[1523] In this way, the present invention improves the efficiency of users' work, and the emotion engine monitors the user's emotional state and prioritizes the automation of stressful tasks, allowing users to work in a more comfortable environment.

[1524] The processing flow will be explained below.

[1525] Step 1:

[1526] Device:

[1527] When a user starts working on their PC, the client software and emotion engine automatically start in the background. The client software records user operation data in real time, such as clicks, keystrokes, application launches and exits, and file operations. At the same time, the emotion engine analyzes the user's facial expressions and tone of voice via the webcam and microphone to identify the user's emotional state in real time.

[1528] Step 2:

[1529] Device:

[1530] The recorded operation data and emotion data are sent to a server at regular intervals, for example, every 30 minutes, and are encrypted to ensure data integrity.

[1531] Step 3:

[1532] server:

[1533] The received operation data and emotion data are stored in a database. The stored data includes the operation details (clicks, input, file operations, etc.) and the user's emotional state (stress, frustration, satisfaction, etc.) along with the date and time.

[1534] Step 4:

[1535] server:

[1536] The stored data is then analyzed using machine learning algorithms. Operational data is used to extract frequent patterns and identify repetitive tasks, while emotional data is used to understand the emotional state a user is experiencing while performing a particular task. For example, it can identify that a user is experiencing high levels of stress while regularly entering data into a spreadsheet.

[1537] Step 5:

[1538] Generated AI (server):

[1539] Based on the identified repetitive tasks and emotion data, the system suggests automation solutions, including script generation and the use of RPA (Robotic Process Automation) tools. Automation solutions are also prioritized for tasks that users find stressful. For example, a script to automate data entry into a spreadsheet is suggested.

[1540] Step 6:

[1541] server:

[1542] Based on the proposed automation measures, specific scripts and configuration files are generated, which include detailed instructions for executing the automation measures, and the generated scripts and configuration files are prepared for deployment to user devices.

[1543] Step 7:

[1544] Device:

[1545] Receives deployed scripts and automation tools from the server, installs them according to the configuration file, and then tests whether the automation tool works properly and adjusts the configuration if necessary.

[1546] Step 8:

[1547] User:

[1548] Run automated tasks and see the results: for example, an automated script instantly inputs data into a spreadsheet, and the emotion engine reduces stress levels.

[1549] Step 9:

[1550] server:

[1551] Regularly monitor the usage and sentiment data of automation tools to evaluate their effectiveness, confirming that automation is improving work efficiency and reducing user frustration, and make additional suggestions or adjust settings as needed.

[1552] Example 2

[1553] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1554] While conventional task automation systems were able to record and analyze user operations, it was difficult to propose automation methods that took the user's emotional state into account. As a result, it was not possible to appropriately prioritize automation for tasks that users found stressful, resulting in insufficient improvement in work efficiency.

[1555] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1556] In this invention, the server includes means for recording user operations, means for analyzing the recorded operations and identifying repetitive tasks, means for proposing automation measures based on the identified repetitive tasks and emotion data, means for installing the proposed automation measures in the user's terminal, and means for transmitting the recorded operation data and emotion data to the server at regular intervals and storing them in a database. This allows the user to appropriately set automation priorities for tasks that cause stress, thereby improving work efficiency.

[1557] "Means for recording user operations" refers to software or hardware that has the ability to record in real time clicks, keystrokes, application startup / exits, file operations, etc., performed by a user on a PC or other device.

[1558] The "means for identifying repetitive tasks" is a component that analyzes recorded user operation data and runs machine learning algorithms or other analytical techniques to identify frequently performed operations or tasks that are repeated based on certain rules.

[1559] An "emotion engine" is a piece of software or hardware that analyzes a user's facial expressions, tone of voice, etc. to identify the user's emotional state in real time. Specifically, it has the function of detecting the user's emotions using a webcam or microphone.

[1560] "Automation suggestion method" refers to an AI model or rule-based system that suggests optimal automation methods to users based on identified repetitive tasks and sentiment data, including script generation and the use of RPA tools.

[1561] "Means for introducing into the user's terminal" refers to the process for installing and configuring the proposed automation means into the user's terminal and introducing it so that it operates normally.

[1562] The "means for transmitting to a server at regular intervals" refers to a system that has the function of transmitting the recorded operation data and emotion data to a server at specific time intervals in batch processing or in real time.

[1563] The "means for saving to a database" is a component that stores the received operation data and emotion data in a database in a format that can be analyzed later. The data is saved in chronological order and used for later analysis and search.

[1564] This invention achieves higher business efficiency by combining a system that automatically identifies repetitive tasks by recording and analyzing user operations, and proposes and implements appropriate automation measures with an emotion engine that recognizes user emotions. This system involves installing client software and an emotion engine on the user's device, and a series of processes that analyze and store operation data and emotion data on a server.

[1565] A specific embodiment of the system is given below.

[1566] System Overview

[1567] The system consists of client software and an emotion engine installed on the user's device, and a server that analyzes and stores data. The client software records operation data such as clicks, keystrokes, application launches and exits, and file operations. The emotion engine analyzes the user's facial expressions and tone of voice via a webcam and microphone to identify their emotional state.

[1568] Collecting Operational and Emotional Data

[1569] User:

[1570] When a user starts working on their PC, the client software and emotion engine automatically start up in the background, recording all of the user's actions and monitoring the user's emotional state in real time.

[1571] Data transmission and storage

[1572] Device:

[1573] The recorded operation data and emotion data are sent to a server at regular intervals. For example, it is possible to set it to send data in batches every hour. This transmission uses an encrypted communication protocol to ensure the security of the data.

[1574] server:

[1575] The received operation data and emotion data are stored in a database in real time. The operation history and emotion history for each user are stored in chronological order and used for later analysis. For example, details such as what application a specific user used on what date and month, and what tasks they felt stressed during are stored.

[1576] Data analysis and identification of repetitive tasks

[1577] server:

[1578] The saved operation data is analyzed using a machine learning algorithm to identify specific operation patterns. Frequently performed operations and tasks that are repeated according to certain rules are identified as repetitive tasks. At the same time, emotional data is analyzed to identify which tasks the user is experiencing stress or dissatisfaction with. For example, if a user enters data into a spreadsheet at the same time every day, emotional data indicating stress while performing that task can be captured.

[1579] Proposal of automation methods

[1580] Generated AI (server):

[1581] The generative AI proposes optimal automation methods based on the identified repetitive tasks and emotion data. These proposals include generating scripts and using RPA (Robotic Process Automation) tools. Automation methods are prioritized for tasks that are particularly stressful for users. Specifically, if the AI ​​determines that a user is repeatedly entering the same data into Excel and is experiencing high levels of stress while doing so, it will propose a script to automate that task.

[1582] Creating and deploying automation tools

[1583] server:

[1584] According to the proposed automation method, specific scripts and configuration files are generated, including Python scripts and configuration files for RPA (Robotic Process Automation) tools, which are then ready to be deployed to the user's devices.

[1585] Device:

[1586] Receive deployed scripts and automation tools from the server, then install and configure them based on the configuration file. After installation, test the automation tools to ensure they work properly and make any necessary adjustments. For example, verify that the generated scripts automatically input data into a spreadsheet.

[1587] Implementing automation and verifying operational efficiency

[1588] User:

[1589] Execute the automated task and check the effect. For example, by automating data entry into a spreadsheet, the previously manual data entry work can be completed instantly. The emotion engine also confirms a reduction in the user's stress level. This allows the time that would have been spent on manual tasks to be used for other tasks. In particular, work can be carried out with less stress, improving overall work efficiency.

[1590] Specific examples

[1591] Example 1: Creating a market analysis report and managing stress

[1592] Device:

[1593] The system records the user's actions as they periodically collect market data and create reports in Excel, while the emotion engine detects stress from the user's facial expressions while creating the reports.

[1594] server:

[1595] The analysis identifies regular and patterned report creation tasks and the stress experienced by users during those tasks.

[1596] Generated AI (server):

[1597] He proposes a script to automate the collection of market data and inputting it into Excel, and determines that this is an urgent need.

[1598] server:

[1599] Generate the proposed script and deploy it to the user's device.

[1600] Device:

[1601] When you run the script, it will automatically retrieve market data and generate an Excel report.

[1602] User:

[1603] The time spent on creating reports is reduced, and stress is reduced. This allows the time saved to be used for other tasks.

[1604] Example 2: Meeting schedule management and emotion monitoring

[1605] Device:

[1606] The system records users' recurring meeting scheduling actions, including setting up meetings using Outlook Calendar and inviting users to meetings. The emotion engine detects when users experience frustration during the scheduling process.

[1607] server:

[1608] Scheduling regular meetings is identified as a repetitive task, and the task proves to be stressful for the user.

[1609] Generated AI (server):

[1610] Suggest a macro or script to automate meeting scheduling in Outlook and determine that automating this task should be a priority.

[1611] server:

[1612] Generate suggested macros and scripts and deploy them to the user's device.

[1613] Device:

[1614] Automated macros and scripts schedule meetings and automatically invite people.

[1615] User:

[1616] Free yourself from manually scheduling meetings, reducing stress and freeing up time for more important work.

[1617] In this way, the present invention improves the efficiency of users' work and realizes the preferential automation of stressful tasks based on the emotion engine.

[1618] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1619] Step 1:

[1620] User:

[1621] The user starts working on the PC.

[1622] Device:

[1623] When a user logs in to their PC and begins work, the client software and emotion engine automatically start in the background. The client software records the user's clicks, keystrokes, application launches and exits, file operations, and other operational data. At the same time, the emotion engine uses the webcam and microphone to analyze the user's facial expressions and tone of voice, identifying and recording their emotional state in real time.

[1624] Input: User operations, facial expressions, and voice data at the start of the task

[1625] Output: Recorded operation data and emotion data

[1626] Step 2:

[1627] Device:

[1628] The recorded operation data and emotion data are sent to a server at regular intervals using an encrypted communication protocol to ensure data security. For example, the data is sent in batches every hour.

[1629] Input: Recorded operation data and emotion data

[1630] Output: Encrypted data sent to the server

[1631] Step 3:

[1632] server:

[1633] The server receives operation data and emotional data and stores it in a database in real time. Each user's operation history and emotional history are managed in chronological order and used for later analysis. Specific operation patterns and emotional changes can be found from the stored data.

[1634] Input: Operation data and emotion data sent to the server

[1635] Output: Operation history and emotion history stored in a database

[1636] Step 4:

[1637] server:

[1638] The saved operation data is analyzed using machine learning algorithms to identify specific operation patterns and repetitive tasks, while emotional data is analyzed to identify which tasks cause stress to the user. For example, if a user enters data into a spreadsheet at the same time every day, the system can identify a pattern of stress when doing so.

[1639] Input: Operational and emotional data stored in a database

[1640] Output: Identified repetitive tasks and emotional state data

[1641] Step 5:

[1642] Generated AI (server):

[1643] The generative AI proposes optimal automation methods based on the identified repetitive tasks and emotion data. This includes generating scripts and using RPA tools. In particular, it prioritizes automation methods for tasks that are likely to cause stress to the user. For example, if it determines that a user is repeatedly entering the same data into Excel and is experiencing high levels of stress while doing so, it proposes a script to automate that task.

[1644] Input: Identified repetitive tasks and emotional state data

[1645] Output: Proposed automation method (script, RPA tool, etc.)

[1646] Step 6:

[1647] server:

[1648] According to the proposed automation method, specific scripts and configuration files are generated, including Python scripts and configuration files for the RPA tool, which are then ready to be deployed to the user's machine.

[1649] Input: Proposed automation method

[1650] Output: Generated scripts and configuration files

[1651] Step 7:

[1652] Device:

[1653] Receive deployed scripts and automation tools from the server, install and configure them, and test the automation tools to ensure they work properly and adjust them as needed. For example, ensure that a generated script that automatically inputs data into a spreadsheet works properly.

[1654] Input: Scripts and automation tools deployed from the server

[1655] Output: Installed and configured automation tools

[1656] Step 8:

[1657] User:

[1658] Implement automated tasks and see the results. For example, automating data entry into a spreadsheet reduces manual work, and the emotion engine reduces user stress levels. This improves operational efficiency and frees up time spent on manual tasks for other important tasks.

[1659] Input: Installed automation tools

[1660] Output: The automated tasks performed and their operational efficiency effects

[1661] (Application example 2)

[1662] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1663] While conventional business efficiency systems can record user operations and automate repetitive tasks, they do not take into account the user's emotions or stress levels, which means that tasks that cause stress to the user are not automated, making it difficult to achieve effective efficiency. Furthermore, in brick-and-mortar stores, many of the tasks that employees perform daily are repetitive and their emotions fluctuate frequently, so efficiency measures that take stress levels into account are required. This leads to a decrease in employee satisfaction and work efficiency, which has a negative impact on the overall performance of the store.

[1664] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1665] In this invention, the server includes a means for recording user operations, a means for analyzing the recorded operations to identify repetitive tasks, a means for identifying the user's emotions, and a means for analyzing the emotional data and adjusting automation priorities to reduce stress. This enables automation that takes into account not only the user's operation data but also their emotional data, enabling the prioritized automation of tasks that are particularly stressful. As a result, employees are freed from stressful tasks and can focus on more creative and satisfying work, improving the operational efficiency and performance of the entire physical store.

[1666] "Means for recording user operations" refers to a function that collects and saves data such as user clicks, keystrokes, application launches and exits, and file operations in real time.

[1667] "Means for analyzing recorded operations and identifying repetitive tasks" refers to a function that analyzes collected operation data using machine learning algorithms, etc., to detect specific operation patterns that are frequently repeated.

[1668] "Means for suggesting automation measures based on identified repetitive tasks" refers to a function that recommends the use of script generation or automation tools based on data on repetitive tasks.

[1669] "Means for introducing the proposed automation measures into the user's device" refers to the ability to install the proposed automation tools or scripts on the user's device and execute the settings.

[1670] The "means for identifying user emotions" is a function that analyzes the user's facial expressions and tone of voice via a webcam or microphone to identify their emotional state.

[1671] "Means for analyzing emotional data and adjusting automation priorities to reduce stress" refers to a function that identifies tasks that cause stress to users based on emotional data, and then suggests and executes the automation of those tasks as a priority.

[1672] The "means for storing the recorded operation data" is a function for periodically sending the collected operation data to a server or database and storing it.

[1673] "Means for learning operation patterns based on stored data" refers to a function that uses stored data to utilize machine learning algorithms, etc., to identify and learn newly emerging patterns and existing patterns.

[1674] "Means for storing emotional data" refers to a function that periodically sends collected emotional data to a server or database and stores it.

[1675] The "means for analyzing stored emotion data and identifying stressful tasks" is a function for analyzing stored emotion data and identifying tasks that cause stress to the user.

[1676] The "means for evaluating the execution frequency of identified tasks and determining automation priorities based on the frequency" is a function that analyzes task execution frequency data and prioritizes automation of tasks that occur frequently.

[1677] The "means for readjusting automation priorities based on stress level" is a function that reviews the initial automation priorities based on the user's stress level data, thereby achieving more effective stress reduction.

[1678] This invention is a system for improving the work efficiency and reducing stress of employees in brick-and-mortar stores. Specific embodiments for realizing this system will be described below.

[1679] System configuration

[1680] The system includes the following hardware and software:

[1681] Smartphones or smart glasses: Equipped with a camera and microphone, they collect user interaction and emotion data.

[1682] EmotionRecognizer class: Analyzes user emotions using an open-source emotion recognition library.

[1683] TaskAutomator class: A custom library for automating tasks.

[1684] Server: Stores operation data and emotion data and performs data analysis using machine learning algorithms.

[1685] Program processing

[1686] The server first launches an application on a smartphone or smart glasses to collect user operation data in real time. The application records the employee's clicks, keystrokes, system startup and shutdown, etc. It also uses the built-in camera and microphone to collect emotional data from facial expressions and tone of voice.

[1687] The collected data is sent to a server at regular intervals, where it is stored in a database of both user operation and emotion data. The server then uses machine learning algorithms to identify repetitive tasks and analyzes the emotion data to identify tasks that cause stress to the user.

[1688] For high-stress, frequently repetitive tasks, the system uses a generative AI model to suggest optimal automation methods, which are then embodied as scripts or automation tools and deployed from a server to users' devices.

[1689] The system installs the proposed automation tool on the user's device and optimizes it based on the configuration file. During this process, the system checks the operation of the automation tool and makes adjustments as necessary.

[1690] Employees only need to perform automated tasks, significantly reducing the amount of manual work they need to do, which increases work efficiency and reduces stress levels.

[1691] Specific examples

[1692] For example, suppose there is an employee who handles accounting at a cash register in a physical store. In this case, the system operates as follows:

[1693] 1. Operation Record and Emotion Recognition

[1694] Record employees scanning items into the cash register system.

[1695] Using the camera in smart glasses or the microphone in a smartphone, it analyzes employees' facial expressions and voice to identify their emotional state.

[1696] 2. Data transmission and analysis

[1697] The collected operation data and emotion data are sent to a server.

[1698] The server stores and analyzes this data.

[1699] 3. Identifying repetitive tasks and suggesting automation

[1700] Identify that cash register operations are repetitive tasks.

[1701] Emotional data can be used to identify stress experienced while operating a cash register.

[1702] As an automation suggestion, we generate scripts to automate product scanning and accounting processes.

[1703] 4. Implementing automation tools

[1704] The scripts proposed by the server are deployed to the user's device and configured.

[1705] 5. Execute automation

[1706] Automated scripts handle cash register operations, leaving employees to simply verify the transaction.

[1707] In this way, the burden on employees can be reduced and work efficiency can be improved.

[1708] Prompt Sentence Examples

[1709] A sample prompt is as follows:

[1710] "Record cashier activity, analyze sentiment data, and suggest tasks that can be automated, such as scanning items or printing receipts."

[1711] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1712] Step 1:

[1713] When a user begins work on a device, an application installed on the smartphone or smart glasses automatically starts in the background. The application records user operation data in real time, including clicks, keystrokes, system startup / shutdown, and file operations. At the same time, it uses the built-in camera and microphone to collect emotional data from facial expressions and tone of voice. As a result, the input data becomes operation data and emotional data. The output data is temporarily stored on the device as separate datasets.

[1714] Step 2:

[1715] The device transmits the recorded operation data and emotion data to the server at regular intervals. The transmission frequency is adjusted based on network load and security considerations, for example, every 30 minutes. The input to this step is the operation data and emotion data recorded in step 1, and the output is the data transmitted to the server.

[1716] Step 3:

[1717] The operation data and emotion data received by the server are immediately stored in a database. The stored data includes a timestamp, operation details, system information of the operation target, and the user's emotional state. The input of this step is the operation data and emotion data sent from the terminal, and the output is storage in the database.

[1718] Step 4:

[1719] The server uses a machine learning algorithm to analyze the stored data and identify specific operation patterns. Repetitive operations and regularly repeated tasks are identified as repetitive tasks. At the same time, it analyzes emotional data to identify which tasks employees experience stress during. The input for this step is the operation data and emotional data stored in the database, and the output is the identified repetitive tasks and stressful tasks.

[1720] Step 5:

[1721] The server proposes automation methods based on the repetitive tasks and emotion data identified using the generative AI model. The generative AI model provides optimal methods for automating tasks, such as script generation and automation tool configuration. It also adjusts automation priorities according to the user's stress level. The inputs for this step are the repetitive tasks and stress data identified in step 4, and the output is specific automation method proposals.

[1722] Step 6:

[1723] The server instantiates the proposed automation measures and generates scripts and configuration files. The generated data is ready to be deployed to the user's device. The input to this step is the automation measures proposed in step 5, and the output is the generated scripts and configuration files.

[1724] Step 7:

[1725] The terminal receives the deployed scripts and automation tools from the server, installs and configures them on the terminal, and performs the necessary tests to verify that the tools work properly. The input to this step is the scripts and configuration files sent from the server, and the output is the installed automation tools.

[1726] Step 8:

[1727] The user executes the automated task and checks its effectiveness. For example, by automating cash register operations, employees are only required to perform the checks. This reduces work time and lowers stress levels. The input for this step is the installed automation tool, and the output is the results of the automation and the streamlined business data.

[1728] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1729] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1730] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1731] [Fourth embodiment]

[1732] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1733] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1734] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1735] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1736] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1737] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1738] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1739] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1740] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1741] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1742] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1743] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1744] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1745] System Overview

[1746] This invention is a system that automatically identifies repetitive tasks by recording and analyzing user operations, and proposes and implements appropriate automation measures. This system involves a series of processes: installing client software on the user's device, and analyzing and saving operation data on a server.

[1747] Program processing

[1748] In the embodiment of the present invention, the following program processing is performed.

[1749] 1. Operation Record

[1750] Device:

[1751] Client software installed on the user's PC records user actions in real time, including clicks, keyboard input, application startup and shutdown, file operations, etc. For example, a user entering data into a spreadsheet is recorded.

[1752] 2. Data transmission and storage

[1753] Device:

[1754] The recorded operation data is periodically sent to a server, including the date, time, and details of the operation.

[1755] server:

[1756] The received data is stored in a database, allowing all operation histories to be managed together and used for later analysis.

[1757] 3. Data analysis and identification of repetitive tasks

[1758] server:

[1759] The stored operational data is analyzed using machine learning algorithms to identify specific operational patterns. Frequently performed operations or tasks that are repeated based on certain rules are identified as repetitive tasks. For example, entering data into a spreadsheet in the same format every day can be identified.

[1760] 4. Proposal of automation methods

[1761] Generated AI (server):

[1762] Based on the identified repetitive tasks, it recommends the best means of automation, which can include generating scripts or using RPA tools. For example, to automate data entry into a spreadsheet, it suggests a data entry script that follows a specific format.

[1763] 5. Creating and implementing automation tools

[1764] server:

[1765] Scripts and configuration files are generated to implement the proposed automation methods, and the settings are configured to be deployed to the user's device.

[1766] Device:

[1767] Install the generated scripts and tools and execute the automation according to your settings. When a user opens a spreadsheet, the configured scripts will automatically populate the data.

[1768] 6. Implementing automation and improving business efficiency

[1769] User:

[1770] Ensure your automation tools are working properly and, if necessary, learn how to use them, which will free up time to focus on creative and strategic tasks rather than manual, repetitive tasks.

[1771] Specific examples

[1772] Example 1: Creating a market analysis report

[1773] Device:

[1774] The user's actions of periodically collecting market data and creating reports in Excel are recorded.

[1775] server:

[1776] Analyze recorded operational data to identify regular and consistent patterns of reporting.

[1777] Generated AI (server):

[1778] Propose a script to automate the collection of market data and inputting it into Excel.

[1779] server:

[1780] Generate the proposed script and deploy it to the user's device.

[1781] Device:

[1782] When the user runs the script, market data is automatically retrieved and an Excel report is automatically created.

[1783] User:

[1784] Time spent on report generation is reduced and can be used for data analysis and decision making.

[1785] Example 2: Managing meeting schedules

[1786] Device:

[1787] It records actions taken by users when they schedule regular meetings, such as setting up a meeting using Outlook Calendar and inviting attendees.

[1788] server:

[1789] As a result of analyzing the operational data, scheduling meetings is identified as a recurring task that is performed periodically.

[1790] Generated AI (server):

[1791] Suggest a macro or script to automate meeting scheduling in Outlook.

[1792] server:

[1793] Generate suggested macros and scripts and prepare them for deployment to users' devices.

[1794] Device:

[1795] Macros and scripts automatically schedule meetings and invite attendees.

[1796] User:

[1797] Free yourself from manually scheduling meetings and use that time to coordinate and execute important projects.

[1798] This frees users from repetitive routine tasks, allowing them to focus on more advanced tasks. The system generates AI specialized for specific business processes, improving business efficiency and optimization.

[1799] The processing flow will be explained below.

[1800] Step 1:

[1801] Device:

[1802] When a user starts working on their PC, the client software automatically launches in the background. The client software records operation data in real time, including clicks, keystrokes, application launches and exits, and file operations. This operation data includes the date and time of the operation, as well as details of the application and file involved.

[1803] Step 2:

[1804] Device:

[1805] The recorded operation data is sent to the server at regular intervals. The frequency of data transmission can be adjusted based on network load and security considerations. For example, it can be set to send operation data in batch processing every hour.

[1806] Step 3:

[1807] server:

[1808] The operation data received by the server is stored in a database in real time. The database stores the operation history for each user in chronological order and is used for later analysis. The stored data includes detailed information about the operation and the application in question.

[1809] Step 4:

[1810] server:

[1811] The stored operation data is analyzed using a machine learning algorithm. The algorithm extracts frequently performed operation patterns from the operation data and identifies tasks that are repeated according to specific rules. For example, an operation pattern in which a user enters data into a spreadsheet at the same time every day can be identified.

[1812] Step 5:

[1813] Generated AI (server):

[1814] Based on the identified repetitive tasks, the generative AI suggests the best way to automate them. These suggestions may include generating scripts or using RPA (Robotic Process Automation) tools. The suggestions are customized for each user, showing the most efficient method.

[1815] Step 6:

[1816] server:

[1817] Based on the proposed automation method, specific scripts and configuration files are generated, which contain detailed instructions for the tasks to be automated, and are then prepared for deployment to the user's devices.

[1818] Step 7:

[1819] Device:

[1820] Receive deployed scripts and automation tools from the server, install and configure them according to the configuration files, test the automation tools to ensure they are working properly, and make adjustments as needed.

[1821] Step 8:

[1822] User:

[1823] Execute the automated task and see the results. For example, by automating data entry into a spreadsheet, a task that was previously done manually can now be done instantly by a script. Users can then use the time saved by automation to focus on other creative tasks.

[1824] Step 9:

[1825] server:

[1826] Regularly monitor the use of automation tools and evaluate their effectiveness, checking whether users are becoming more efficient and making additional suggestions or adjustments if further improvements are needed.

[1827] Through the above series of processing steps, users are freed from repetitive routine tasks and can concentrate on creative work.

[1828] Example 1

[1829] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1830] In modern work environments, there are many repetitive tasks that users perform repeatedly. However, these tasks are often performed manually, significantly reducing efficiency. It is also difficult for users to determine which tasks should be automated, and selecting and implementing appropriate automation tools is time-consuming. Therefore, there is a need for a system that records user operations, automatically identifies repetitive tasks, and proposes and implements the optimal automation methods.

[1831] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1832] In this invention, the server includes means for recording user operations in real time, means for periodically transmitting the recorded operation data to the server, means for storing the received data in a database, means for analyzing the stored data using a machine learning algorithm to identify repetitive tasks, means for proposing appropriate automation measures based on the identified repetitive tasks, means for generating scripts or tools that implement the proposed automation measures and installing them on the user's terminal, and means for verifying that the automation tools operate normally on the user's terminal. This frees users from repetitive tasks, allowing them to use their time efficiently and focus on more valuable tasks.

[1833] "User operations" refer to actions such as clicking, keyboard input, starting and closing applications, and file operations that a user performs on a device.

[1834] "Real-time recording means" refers to software or a system that instantly records user operations and saves the data.

[1835] "Operation data" is digital data that includes the content, date and time, details of a user's operation, and information about the application being used.

[1836] The "means for transmitting to the server" refers to a communication protocol or program for transmitting the operation data recorded on the terminal to the server.

[1837] The "means for storing in a database" refers to a system or method for storing the received operation data in a database in a form that allows it to be uniquely identified.

[1838] A "machine learning algorithm" is an algorithm or model that analyzes stored operation data and identifies specific operation patterns.

[1839] A "repetitive task" is an operation or task that is performed in the same sequence and is repeated at regular intervals.

[1840] An "automation tool" is a tool, software, script, or system used to perform repetitive tasks automatically.

[1841] A "script or tool generating means" is a system or software for generating code or configuration files to implement the proposed automation means.

[1842] The "means for introducing the script or tool into the user's terminal" refers to a method or process for placing the generated script or tool on the user's terminal and running it properly.

[1843] "Means to verify proper operation" refers to the process or method for verifying that the implemented scripts and tools operate as intended.

[1844] MODE FOR CARRYING OUT THE INVENTION

[1845] The present invention is a system that automatically identifies repetitive tasks by recording and analyzing user operations, and then proposes and implements appropriate automation measures. This system installs client software on the user's terminal, and analyzes and stores data on a server. Specific embodiments of the system are described below.

[1846] System Overview

[1847] The system records user operations in real time, transmits the operation data to a server, and stores the received data in a database. It uses a machine learning algorithm to analyze operation patterns and identify repetitive tasks. Based on the identified tasks, it proposes automation methods and includes a means for implementing them on the user's device.

[1848] Hardware and software used

[1849] Device:

[1850] Client software is installed on the user's PC, and this software has the function of recording user operations in real time and periodically sending the data to the server.

[1851] server:

[1852] The server stores the received operation data in a database and analyzes it using machine learning algorithms, such as TensorFlow, a machine learning library. Based on the analysis results, it proposes automation methods, generates scripts and tools, and deploys them to the user's device.

[1853] Database:

[1854] The database stores operational data in a uniquely identifiable form, which is then used for later analysis by machine learning algorithms.

[1855] Specific actions

[1856] User operation record:

[1857] The client software on the device records user actions such as clicks, keyboard input, and application launches and exits, generating detailed operational data.

[1858] Data transmission and storage:

[1859] The recorded operation data is sent to a server, for example, every five minutes, and the server stores the received data in a database for later analysis.

[1860] Data analysis and automation methods proposed:

[1861] The server analyzes the stored data using machine learning algorithms to identify repetitive tasks, and then suggests the best automation method (script or tool) based on the identified tasks.

[1862] Generate and deploy scripts and tools:

[1863] The proposed automation methods are embodied as scripts or tools and deployed to the user's terminals, for example, Python scripts or RPA tools.

[1864] Check the operation of the automation tool:

[1865] Users ensure that automated scripts and tools work properly, automating repetitive tasks and improving work efficiency.

[1866] Specific examples

[1867] Example 1: Creating a market analysis report

[1868] Device:

[1869] This tool records the actions of users who periodically collect market data and create reports in Excel. It also records the actions of downloading data from market data sites and entering it into Excel sheets.

[1870] server:

[1871] The recorded operation data is analyzed and "downloading market data" and "entering it into an Excel spreadsheet" are identified as repetitive tasks.

[1872] Generated AI (server):

[1873] We propose a Python script to automate the collection of market data and inputting it into Excel.

[1874] server:

[1875] Generates the proposed Python script and deploys it to the user's device.

[1876] Device:

[1877] When the user runs the script, market data is automatically retrieved and an Excel report is automatically generated.

[1878] User:

[1879] Spend less time creating reports and more time analyzing data and making decisions.

[1880] Prompt Sentence Examples

[1881] "Please suggest a script to record and automate the regular market data collection and Excel report creation."

[1882] Example 2: Managing meeting schedules

[1883] Device:

[1884] This records the actions of users who schedule regular meetings, set up meetings in their Outlook calendar, and invite attendees.

[1885] server:

[1886] By analyzing the operation data, "scheduling a meeting" and "inviting attendees" are identified as repetitive tasks.

[1887] Generated AI (server):

[1888] Suggest a VBA macro or Python script to automate meeting scheduling in Outlook.

[1889] server:

[1890] Generate suggested macros and scripts and prepare them for deployment to users' devices.

[1891] Device:

[1892] Macros and scripts automatically schedule meetings and invite attendees.

[1893] User:

[1894] Free yourself from the manual work of scheduling meetings and use that time to coordinate and execute important projects.

[1895] Prompt Sentence Examples

[1896] "Please suggest a macro or script that records and automates routine meeting setup and attendee invitation actions."

[1897] This frees users from repetitive tasks, allowing them to focus on more creative and strategic work.

[1898] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1899] Step 1:

[1900] Operation Record

[1901] Device:

[1902] Client software installed on the user's PC records the user's actions in real time. Specifically, each time the user clicks, the location information, the details of the action (e.g., button click, form entry), and the application being used are captured. For example, the user entering "sales data" and setting a filter in a spreadsheet is recorded.

[1903] input:

[1904] User actions (clicks, keyboard input, etc.)

[1905] output:

[1906] Operation data (location information, action details, application information, etc.)

[1907] Step 2:

[1908] Data transmission

[1909] Device:

[1910] At regular intervals (for example, every 5 minutes), the recorded operation data is sent to the server. The sent data includes the date and time of the operation, detailed content, and information about the application used.

[1911] input:

[1912] Recorded operation data

[1913] output:

[1914] Operation data sent to the server

[1915] Step 3:

[1916] Data storage

[1917] server:

[1918] The received data is saved in a database. An ID and timestamp are assigned to the database so that operation history can be stored in a uniquely identifiable form. For example, the operation "Enter sales data into a spreadsheet" performed on "2023-01-01 12:00:00" is saved as "Operation History ID: 12345."

[1919] input:

[1920] Operation data sent to the server

[1921] output:

[1922] Operational data stored in a database

[1923] Step 4:

[1924] Data analysis

[1925] server:

[1926] The stored data is analyzed using machine learning algorithms. For example, machine learning libraries such as TensorFlow are used. Through the analysis, frequently performed operations or tasks that are repeated based on certain rules are identified as "repetitive tasks." For example, it may be detected that "entering sales data into a spreadsheet" is performed every day.

[1927] input:

[1928] Operational data stored in a database

[1929] output:

[1930] Repetitive task identification results

[1931] Step 5:

[1932] Proposal of automation methods

[1933] Generated AI (server):

[1934] Based on the analysis results, the system proposes optimal automation methods for identified repetitive tasks. Specifically, it generates scripts or bots (e.g., Python scripts or RPA tools) that mimic the operational procedures and suggests them to the user. For example, it proposes generating a Python script that automatically inputs sales data into a spreadsheet.

[1935] input:

[1936] Repetitive task identification results

[1937] output:

[1938] Proposal of automation methods (scripts and tool specifications)

[1939] Step 6:

[1940] Generating scripts and tools

[1941] server:

[1942] Generate scripts and configuration files to implement the proposed automation measures, for example, generate the proposed Python scripts and package them into a ZIP file.

[1943] input:

[1944] Proposal of automation methods (scripts and tool specifications)

[1945] output:

[1946] Concretized scripts and tools (e.g. generated Python scripts)

[1947] Step 7:

[1948] Deploying scripts and tools

[1949] server:

[1950] Prepare the generated scripts and tools for deployment to the user's device and perform the necessary configuration.

[1951] Device:

[1952] Install the generated scripts and tools on the terminal and register them in the task scheduler to run at 10:00 every day.

[1953] input:

[1954] Embedding scripts and tools (ZIP files, etc.)

[1955] output:

[1956] Scripts and tools deployed on user devices

[1957] Step 8:

[1958] Operation check

[1959] User:

[1960] Ensure automated scripts and tools are working properly and, if necessary, learn how to use them. Well-working scripts automate repetitive tasks and improve work efficiency.

[1961] input:

[1962] Scripts and tools deployed on user devices

[1963] output:

[1964] Confirmed working automation tools

[1965] (Application example 1)

[1966] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1967] The large amount of repetitive manual work at logistics centers consumes a lot of labor and time, resulting in issues such as reduced productivity and increased human error. In particular, efficient automation is required for routine tasks such as picking and inventory taking. Automating these tasks will improve operational efficiency and reduce the burden on workers.

[1968] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1969] In this invention, the server includes means for recording user operations, means for analyzing the recorded operations and identifying repetitive tasks, means for proposing automation measures based on the identified repetitive tasks, means for installing the proposed automation measures on the user's terminal, means for transmitting and storing operation data recorded in logistics operations to a cloud server, means for analyzing the stored data using a machine learning algorithm and identifying repetitive work patterns, and means for generating automation scripts based on the identified work patterns and installing them on the user's mobile device. This makes it possible to efficiently automate repetitive routine tasks performed in logistics centers, reduce the burden on workers, and improve productivity.

[1970] "User" means a person who operates the System or an end user.

[1971] "Means for recording operations" refers to software or devices for capturing and saving user operations in real time.

[1972] A "repetitive task" is work that follows a set pattern, where the same operations are repeated over and over again.

[1973] "Automation tools" are scripts and tools proposed to automate repetitive user tasks.

[1974] "Terminal deployment" refers to the process of installing or making executable the automation means on the user's device.

[1975] A "cloud server" is an external server that stores and analyzes data via the Internet.

[1976] A "machine learning algorithm" is an algorithm that learns patterns from large amounts of data and makes predictions and classifications for new data.

[1977] A "work pattern" is a repetitive action or sequence of operations performed by a user.

[1978] An "automation script" is a program or code created to perform a specific task automatically.

[1979] A "mobile device" is a portable electronic device such as a smartphone or tablet.

[1980] "Storage means" refers to a memory or database for reliably storing recorded data.

[1981] "Proposed means" are functions or processes that show users the automated means.

[1982] "Means for analysis" refers to the systems and processes used to analyze the recorded operational data and extract meaningful information.

[1983] This invention is a system that aims to automate repetitive tasks in logistics centers and maximize operational efficiency. Specifically, it records user operations in real time and analyzes them using machine learning algorithms to identify repetitive tasks and propose and implement automation methods.

[1984] Overall system configuration

[1985] 1. A means of recording user actions

[1986] The client application installed on the device records logistics operations in real time, such as QR code scanners and touchscreen input, including clicks, keyboard input, application launches and exits, and file operations.

[1987] 2. Means of transmitting and storing recorded operating data

[1988] The operation data recorded on the device is periodically sent to a cloud server and stored in AWS or a database (SQLite, PostgreSQL, etc.). This data includes the date and time of the operation and details.

[1989] 3. Data analysis and identification of repetitive tasks

[1990] The server analyzes the stored operation data using machine learning algorithms (e.g., Python, TensorFlow, etc.), thereby identifying frequently performed operations and tasks that are repeated based on certain rules.

[1991] 4. Proposal of automation methods

[1992] The server uses a generative AI model to suggest optimal automation strategies based on identified repetitive tasks, taking into account the frequency and priority of the tasks. For example, a script may be generated for a specific item picking or periodic inventory task.

[1993] 5. Creating and implementing automation tools

[1994] Based on the proposed automation measures, scripts and configuration files are generated and deployed to users' mobile devices (smartphones and tablets). The scripts automate certain repetitive tasks, such as automatically updating new inventory data.

[1995] 6. Automation and operational efficiency

[1996] Using the generated automation tools, users can streamline logistics operations. By automating picking and inventory tasks, the burden on workers is reduced, allowing them to spend more time focusing on important tasks.

[1997] Specific examples

[1998] Example 1: Automated product picking

[1999] At a logistics center, the system records the product picking work performed by workers every day. The data sent to a cloud server is analyzed and repetitive picking patterns are identified. Based on the results, a script to automate the picking work is generated and deployed to smartphones. Workers can use their smartphones to automate picking work, allowing them to work more efficiently.

[2000] Example prompts to input to a generative AI model:

[2001] Generate a script to automate the daily picking tasks performed by distribution center workers. You analyzed the following operational data and identified repetitive picking patterns. Provide an example of a script that would automate this task.

[2002] This system efficiently automates repetitive routine tasks at logistics centers, significantly reducing the burden on workers. By utilizing a combination of generative AI models and machine learning algorithms, it achieves advanced operational efficiency.

[2003] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2004] Step 1:

[2005] When working in the distribution center, users use a client application on their smartphones that uses a QR code scanner or touch screen to record operational data such as picking and inventory in real time.

[2006] Input: Product identification information and work instructions

[2007] Output: Data recording the type and time of operation

[2008] Specific operation: Scan the QR code and save the operation details and timestamp to local storage.

[2009] Step 2:

[2010] The device periodically transmits the recorded data to the cloud server, including detailed information about the operation, the date and time, and information about the working environment.

[2011] Input: Operation data stored in local storage

[2012] Output: Data transferred to the cloud server

[2013] What happens: A scheduled job collects data and sends it over the internet to a cloud server.

[2014] Step 3:

[2015] The server stores the received operational data in a database, where it is formatted and preprocessed for analysis.

[2016] Input: Operation data sent

[2017] Output: Formatted operation data stored in the database

[2018] What happens: The server's receiving module parses the data and imports it into the database in the appropriate format.

[2019] Step 4:

[2020] The server runs machine learning algorithms on the stored operational data to identify repetitive work patterns, using algorithms such as Python and TensorFlow.

[2021] Input: Operational data in the database

[2022] Output: A list of identified recurring tasks

[2023] What it does: Train and evaluate machine learning models to extract important patterns.

[2024] Step 5:

[2025] The server uses a generative AI model to suggest automation measures based on the identified repetitive tasks, and the suggested automation measures are generated in the form of scripts.

[2026] Input: A list of identified recurring tasks

[2027] Output: Automation script

[2028] Specific operation: Provide input data to the generative AI model and obtain the generated script.

[2029] Step 6:

[2030] The proposed automation script is deployed to the device, where the user receives the automation script and prepares it to run on the device.

[2031] Input: Automation script

[2032] Output: Script delivered to the terminal

[2033] Specific operation: The server sends the script to the device, and the application on the device installs or updates it.

[2034] Step 7:

[2035] Users run automation scripts on their devices, which automatically process designated tasks and re-record operation data, improving work efficiency.

[2036] Input: Installed automation script

[2037] Output: The result of the automated task

[2038] What it does: The user launches the script, and the device automatically performs the task and displays the results.

[2039] Through this series of steps, repetitive tasks within the logistics center can be automated, dramatically improving work efficiency.

[2040] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2041] System Overview

[2042] This invention achieves higher business efficiency by combining a system that automatically identifies repetitive tasks by recording and analyzing user operations and proposes and implements appropriate automation measures with an emotion engine that recognizes user emotions. This system involves installing client software and an emotion engine on the user's device and a series of processes that analyze and store operation data and emotion data on a server.

[2043] Program processing

[2044] In the embodiment of the present invention, the following program processing is performed.

[2045] 1. Operation Record and Emotion Recognition

[2046] Device:

[2047] When a user starts working on their PC, the client software and emotion engine automatically start in the background. The client software records operation data such as clicks, keystrokes, application startup / exit, and file operations in real time. At the same time, the emotion engine analyzes the user's facial expressions and tone of voice via the webcam and microphone to identify the user's emotional state.

[2048] 2. Data transmission and storage

[2049] Device:

[2050] The recorded operation data and emotion data are sent to the server at regular intervals. The frequency of data transmission can be adjusted based on network load and security considerations. For example, it can be set to send operation data and emotion data in batch processing every hour.

[2051] server:

[2052] The received operation data and emotion data are stored in a database in real time. The database stores each user's operation history and emotion history in chronological order and is used for later analysis. The stored data includes details of the operation, the target application, and the user's emotional state.

[2053] 3. Data analysis and identification of repetitive tasks

[2054] server:

[2055] The system analyzes the saved operation data using a machine learning algorithm to identify specific operation patterns. Frequently performed operations and tasks that are repeated according to certain rules are identified as repetitive tasks. At the same time, it analyzes emotional data to identify which tasks cause stress or frustration for the user. For example, while identifying an operation pattern in which a user inputs data into a spreadsheet at the same time every day, it also captures emotional data indicating the user is frustrated while performing the task.

[2056] 4. Proposal of automation methods

[2057] Generated AI (server):

[2058] The generative AI then proposes optimal automation methods based on the identified repetitive tasks and emotional data. These proposals include script generation and the use of RPA (Robotic Process Automation) tools. Automation methods are prioritized for tasks that are likely to cause stress to users. For example, a script for automating data entry into a spreadsheet is proposed, and emotional data indicates that automation is urgently needed.

[2059] 5. Creating and implementing automation tools

[2060] server:

[2061] Based on the proposed automation method, specific scripts and configuration files are generated, which contain detailed instructions for the tasks to be automated, and are then prepared for deployment to the user's devices.

[2062] Device:

[2063] Receive deployed scripts and automation tools from the server, install and configure them according to the configuration files, test the automation tools to ensure they are working properly, and make adjustments as needed.

[2064] 6. Implementing automation and improving business efficiency

[2065] User:

[2066] Execute the automated task and observe the effect. For example, by automating data entry into a spreadsheet, a task that was previously done manually can now be done instantly by a script. Also, observe that the emotion engine reduces the user's stress level. Users can use the time saved by automation to pursue other creative tasks.

[2067] Specific examples

[2068] Example 1: Creating a market analysis report and managing stress

[2069] Device:

[2070] The system records the user's actions as they periodically collect market data and create reports in Excel. At the same time, an emotion engine detects whether the user is feeling stressed from their facial expressions while creating the reports.

[2071] server:

[2072] The recorded operation data and emotion data are analyzed to identify whether report creation is occurring regularly and in the same pattern, and to identify that the user is feeling stressed while performing the task.

[2073] Generated AI (server):

[2074] They propose a script to automate the collection of market data and inputting it into Excel, and determine that automation is urgently needed based on sentiment data.

[2075] server:

[2076] Generate the proposed script and deploy it to the user's device.

[2077] Device:

[2078] When the user runs the script, market data is automatically retrieved and an Excel report is automatically created.

[2079] User:

[2080] Time spent on report creation is reduced and can be spent on data analysis and decision-making, reducing stress and improving work efficiency.

[2081] Example 2: Meeting schedule management and emotion monitoring

[2082] Device:

[2083] The system records the user's actions to schedule regular meetings, such as setting up meetings using Outlook Calendar and inviting attendees. The emotion engine also detects frustration from the user's facial expressions during the meeting setup process.

[2084] server:

[2085] Analysis of the operation data identifies scheduling a meeting as a repetitive task that is performed periodically, and emotion data reveals that the task is stressful for the user.

[2086] Generated AI (server):

[2087] Suggest a macro or script to automate meeting scheduling in Outlook, and determine based on sentiment data that automation of this task should be a priority.

[2088] server:

[2089] Generate suggested macros and scripts and prepare them for deployment to users' devices.

[2090] Device:

[2091] Macros and scripts automatically schedule meetings and invite attendees.

[2092] User:

[2093] This frees employees from manually scheduling meetings and allows them to focus on coordinating and implementing important projects. Emotional data helps reduce stress and improve overall work efficiency.

[2094] In this way, the present invention improves the efficiency of users' work, and the emotion engine monitors the user's emotional state and prioritizes the automation of stressful tasks, allowing users to work in a more comfortable environment.

[2095] The processing flow will be explained below.

[2096] Step 1:

[2097] Device:

[2098] When a user starts working on their PC, the client software and emotion engine automatically start in the background. The client software records user operation data in real time, such as clicks, keystrokes, application launches and exits, and file operations. At the same time, the emotion engine analyzes the user's facial expressions and tone of voice via the webcam and microphone to identify the user's emotional state in real time.

[2099] Step 2:

[2100] Device:

[2101] The recorded operation data and emotion data are sent to a server at regular intervals, for example, every 30 minutes, and are encrypted to ensure data integrity.

[2102] Step 3:

[2103] server:

[2104] The received operation data and emotion data are stored in a database. The stored data includes the operation details (clicks, input, file operations, etc.) and the user's emotional state (stress, frustration, satisfaction, etc.) along with the date and time.

[2105] Step 4:

[2106] server:

[2107] The stored data is then analyzed using machine learning algorithms. Operational data is used to extract frequent patterns and identify repetitive tasks, while emotional data is used to understand the emotional state a user is experiencing while performing a particular task. For example, it can identify that a user is experiencing high levels of stress while regularly entering data into a spreadsheet.

[2108] Step 5:

[2109] Generated AI (server):

[2110] Based on the identified repetitive tasks and emotion data, the system suggests automation solutions, including script generation and the use of RPA (Robotic Process Automation) tools. Automation solutions are also prioritized for tasks that users find stressful. For example, a script to automate data entry into a spreadsheet is suggested.

[2111] Step 6:

[2112] server:

[2113] Based on the proposed automation measures, specific scripts and configuration files are generated, which include detailed instructions for executing the automation measures, and the generated scripts and configuration files are prepared for deployment to user devices.

[2114] Step 7:

[2115] Device:

[2116] Receives deployed scripts and automation tools from the server, installs them according to the configuration file, and then tests whether the automation tool works properly and adjusts the configuration if necessary.

[2117] Step 8:

[2118] User:

[2119] Run automated tasks and see the results: for example, an automated script instantly inputs data into a spreadsheet, and the emotion engine reduces stress levels.

[2120] Step 9:

[2121] server:

[2122] Regularly monitor the usage and sentiment data of automation tools to evaluate their effectiveness, confirming that automation is improving work efficiency and reducing user frustration, and make additional suggestions or adjust settings as needed.

[2123] Example 2

[2124] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2125] While conventional task automation systems were able to record and analyze user operations, it was difficult to propose automation methods that took the user's emotional state into account. As a result, it was not possible to appropriately prioritize automation for tasks that users found stressful, resulting in insufficient improvement in work efficiency.

[2126] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2127] In this invention, the server includes means for recording user operations, means for analyzing the recorded operations and identifying repetitive tasks, means for proposing automation measures based on the identified repetitive tasks and emotion data, means for installing the proposed automation measures in the user's terminal, and means for transmitting the recorded operation data and emotion data to the server at regular intervals and storing them in a database. This allows the user to appropriately set automation priorities for tasks that cause stress, thereby improving work efficiency.

[2128] "Means for recording user operations" refers to software or hardware that has the ability to record in real time clicks, keystrokes, application startup / exits, file operations, etc., performed by a user on a PC or other device.

[2129] The "means for identifying repetitive tasks" is a component that analyzes recorded user operation data and runs machine learning algorithms or other analytical techniques to identify frequently performed operations or tasks that are repeated based on certain rules.

[2130] An "emotion engine" is a piece of software or hardware that analyzes a user's facial expressions, tone of voice, etc. to identify the user's emotional state in real time. Specifically, it has the function of detecting the user's emotions using a webcam or microphone.

[2131] "Automation suggestion method" refers to an AI model or rule-based system that suggests optimal automation methods to users based on identified repetitive tasks and sentiment data, including script generation and the use of RPA tools.

[2132] "Means for introducing into the user's terminal" refers to the process for installing and configuring the proposed automation means into the user's terminal and introducing it so that it operates normally.

[2133] The "means for transmitting to a server at regular intervals" refers to a system that has the function of transmitting the recorded operation data and emotion data to a server at specific time intervals in batch processing or in real time.

[2134] The "means for saving to a database" is a component that stores the received operation data and emotion data in a database in a format that can be analyzed later. The data is saved in chronological order and used for later analysis and search.

[2135] This invention achieves higher business efficiency by combining a system that automatically identifies repetitive tasks by recording and analyzing user operations, and proposes and implements appropriate automation measures with an emotion engine that recognizes user emotions. This system involves installing client software and an emotion engine on the user's device, and a series of processes that analyze and store operation data and emotion data on a server.

[2136] A specific embodiment of the system is given below.

[2137] System Overview

[2138] The system consists of client software and an emotion engine installed on the user's device, and a server that analyzes and stores data. The client software records operation data such as clicks, keystrokes, application launches and exits, and file operations. The emotion engine analyzes the user's facial expressions and tone of voice via a webcam and microphone to identify their emotional state.

[2139] Collecting Operational and Emotional Data

[2140] User:

[2141] When a user starts working on their PC, the client software and emotion engine automatically start up in the background, recording all of the user's actions and monitoring the user's emotional state in real time.

[2142] Data transmission and storage

[2143] Device:

[2144] The recorded operation data and emotion data are sent to a server at regular intervals. For example, it is possible to set it to send data in batches every hour. This transmission uses an encrypted communication protocol to ensure the security of the data.

[2145] server:

[2146] The received operation data and emotion data are stored in a database in real time. The operation history and emotion history for each user are stored in chronological order and used for later analysis. For example, details such as what application a specific user used on what date and month, and what tasks they felt stressed during are stored.

[2147] Data analysis and identification of repetitive tasks

[2148] server:

[2149] The saved operation data is analyzed using a machine learning algorithm to identify specific operation patterns. Frequently performed operations and tasks that are repeated according to certain rules are identified as repetitive tasks. At the same time, emotional data is analyzed to identify which tasks the user is experiencing stress or dissatisfaction with. For example, if a user enters data into a spreadsheet at the same time every day, emotional data indicating stress while performing that task can be captured.

[2150] Proposal of automation methods

[2151] Generated AI (server):

[2152] The generative AI proposes optimal automation methods based on the identified repetitive tasks and emotion data. These proposals include generating scripts and using RPA (Robotic Process Automation) tools. Automation methods are prioritized for tasks that are particularly stressful for users. Specifically, if the AI ​​determines that a user is repeatedly entering the same data into Excel and is experiencing high levels of stress while doing so, it will propose a script to automate that task.

[2153] Creating and deploying automation tools

[2154] server:

[2155] According to the proposed automation method, specific scripts and configuration files are generated, including Python scripts and configuration files for RPA (Robotic Process Automation) tools, which are then ready to be deployed to the user's devices.

[2156] Device:

[2157] Receive deployed scripts and automation tools from the server, then install and configure them based on the configuration file. After installation, test the automation tools to ensure they work properly and make any necessary adjustments. For example, verify that the generated scripts automatically input data into a spreadsheet.

[2158] Implementing automation and verifying operational efficiency

[2159] User:

[2160] Execute the automated task and check the effect. For example, by automating data entry into a spreadsheet, the previously manual data entry work can be completed instantly. The emotion engine also confirms a reduction in the user's stress level. This allows the time that would have been spent on manual tasks to be used for other tasks. In particular, work can be carried out with less stress, improving overall work efficiency.

[2161] Specific examples

[2162] Example 1: Creating a market analysis report and managing stress

[2163] Device:

[2164] The system records the user's actions as they periodically collect market data and create reports in Excel, while the emotion engine detects stress from the user's facial expressions while creating the reports.

[2165] server:

[2166] The analysis identifies regular and patterned report creation tasks and the stress experienced by users during those tasks.

[2167] Generated AI (server):

[2168] He proposes a script to automate the collection of market data and inputting it into Excel, and determines that this is an urgent need.

[2169] server:

[2170] Generate the proposed script and deploy it to the user's device.

[2171] Device:

[2172] When you run the script, it will automatically retrieve market data and generate an Excel report.

[2173] User:

[2174] The time spent on creating reports is reduced, and stress is reduced. This allows the time saved to be used for other tasks.

[2175] Example 2: Meeting schedule management and emotion monitoring

[2176] Device:

[2177] The system records users' recurring meeting scheduling actions, including setting up meetings using Outlook Calendar and inviting users to meetings. The emotion engine detects when users experience frustration during the scheduling process.

[2178] server:

[2179] Scheduling regular meetings is identified as a repetitive task, and the task proves to be stressful for the user.

[2180] Generated AI (server):

[2181] Suggest a macro or script to automate meeting scheduling in Outlook and determine that automating this task should be a priority.

[2182] server:

[2183] Generate suggested macros and scripts and deploy them to the user's device.

[2184] Device:

[2185] Automated macros and scripts schedule meetings and automatically invite people.

[2186] User:

[2187] Free yourself from manually scheduling meetings, reducing stress and freeing up time for more important work.

[2188] In this way, the present invention improves the efficiency of users' work and realizes the preferential automation of stressful tasks based on the emotion engine.

[2189] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2190] Step 1:

[2191] User:

[2192] The user starts working on the PC.

[2193] Device:

[2194] When a user logs in to their PC and begins work, the client software and emotion engine automatically start in the background. The client software records the user's clicks, keystrokes, application launches and exits, file operations, and other operational data. At the same time, the emotion engine uses the webcam and microphone to analyze the user's facial expressions and tone of voice, identifying and recording their emotional state in real time.

[2195] Input: User operations, facial expressions, and voice data at the start of the task

[2196] Output: Recorded operation data and emotion data

[2197] Step 2:

[2198] Device:

[2199] The recorded operation data and emotion data are sent to a server at regular intervals using an encrypted communication protocol to ensure data security. For example, the data is sent in batches every hour.

[2200] Input: Recorded operation data and emotion data

[2201] Output: Encrypted data sent to the server

[2202] Step 3:

[2203] server:

[2204] The server receives operation data and emotional data and stores it in a database in real time. Each user's operation history and emotional history are managed in chronological order and used for later analysis. Specific operation patterns and emotional changes can be found from the stored data.

[2205] Input: Operation data and emotion data sent to the server

[2206] Output: Operation history and emotion history stored in a database

[2207] Step 4:

[2208] server:

[2209] The saved operation data is analyzed using machine learning algorithms to identify specific operation patterns and repetitive tasks, while emotional data is analyzed to identify which tasks cause stress to the user. For example, if a user enters data into a spreadsheet at the same time every day, the system can identify a pattern of stress when doing so.

[2210] Input: Operational and emotional data stored in a database

[2211] Output: Identified repetitive tasks and emotional state data

[2212] Step 5:

[2213] Generated AI (server):

[2214] The generative AI proposes optimal automation methods based on the identified repetitive tasks and emotion data. This includes generating scripts and using RPA tools. In particular, it prioritizes automation methods for tasks that are likely to cause stress to the user. For example, if it determines that a user is repeatedly entering the same data into Excel and is experiencing high levels of stress while doing so, it proposes a script to automate that task.

[2215] Input: Identified repetitive tasks and emotional state data

[2216] Output: Proposed automation method (script, RPA tool, etc.)

[2217] Step 6:

[2218] server:

[2219] According to the proposed automation method, specific scripts and configuration files are generated, including Python scripts and configuration files for the RPA tool, which are then ready to be deployed to the user's machine.

[2220] Input: Proposed automation method

[2221] Output: Generated scripts and configuration files

[2222] Step 7:

[2223] Device:

[2224] Receive deployed scripts and automation tools from the server, install and configure them, and test the automation tools to ensure they work properly and adjust them as needed. For example, ensure that a generated script that automatically inputs data into a spreadsheet works properly.

[2225] Input: Scripts and automation tools deployed from the server

[2226] Output: Installed and configured automation tools

[2227] Step 8:

[2228] User:

[2229] Implement automated tasks and see the results. For example, automating data entry into a spreadsheet reduces manual work, and the emotion engine reduces user stress levels. This improves operational efficiency and frees up time spent on manual tasks for other important tasks.

[2230] Input: Installed automation tools

[2231] Output: The automated tasks performed and their operational efficiency effects

[2232] (Application example 2)

[2233] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2234] While conventional business efficiency systems can record user operations and automate repetitive tasks, they do not take into account the user's emotions or stress levels, which means that tasks that cause stress to the user are not automated, making it difficult to achieve effective efficiency. Furthermore, in brick-and-mortar stores, many of the tasks that employees perform daily are repetitive and their emotions fluctuate frequently, so efficiency measures that take stress levels into account are required. This leads to a decrease in employee satisfaction and work efficiency, which has a negative impact on the overall performance of the store.

[2235] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2236] In this invention, the server includes a means for recording user operations, a means for analyzing the recorded operations to identify repetitive tasks, a means for identifying the user's emotions, and a means for analyzing the emotional data and adjusting automation priorities to reduce stress. This enables automation that takes into account not only the user's operation data but also their emotional data, enabling the prioritized automation of tasks that are particularly stressful. As a result, employees are freed from stressful tasks and can focus on more creative and satisfying work, improving the operational efficiency and performance of the entire physical store.

[2237] "Means for recording user operations" refers to a function that collects and saves data such as user clicks, keystrokes, application launches and exits, and file operations in real time.

[2238] "Means for analyzing recorded operations and identifying repetitive tasks" refers to a function that analyzes collected operation data using machine learning algorithms, etc., to detect specific operation patterns that are frequently repeated.

[2239] "Means for suggesting automation measures based on identified repetitive tasks" refers to a function that recommends the use of script generation or automation tools based on data on repetitive tasks.

[2240] "Means for introducing the proposed automation measures into the user's device" refers to the ability to install the proposed automation tools or scripts on the user's device and execute the settings.

[2241] The "means for identifying user emotions" is a function that analyzes the user's facial expressions and tone of voice via a webcam or microphone to identify their emotional state.

[2242] "Means for analyzing emotional data and adjusting automation priorities to reduce stress" refers to a function that identifies tasks that cause stress to users based on emotional data, and then suggests and executes the automation of those tasks as a priority.

[2243] The "means for storing the recorded operation data" is a function for periodically sending the collected operation data to a server or database and storing it.

[2244] "Means for learning operation patterns based on stored data" refers to a function that uses stored data to utilize machine learning algorithms, etc., to identify and learn newly emerging patterns and existing patterns.

[2245] "Means for storing emotional data" refers to a function that periodically sends collected emotional data to a server or database and stores it.

[2246] The "means for analyzing stored emotion data and identifying stressful tasks" is a function for analyzing stored emotion data and identifying tasks that cause stress to the user.

[2247] The "means for evaluating the execution frequency of identified tasks and determining automation priorities based on the frequency" is a function that analyzes task execution frequency data and prioritizes automation of tasks that occur frequently.

[2248] The "means for readjusting automation priorities based on stress level" is a function that reviews the initial automation priorities based on the user's stress level data, thereby achieving more effective stress reduction.

[2249] This invention is a system for improving the work efficiency and reducing stress of employees in brick-and-mortar stores. Specific embodiments for realizing this system will be described below.

[2250] System configuration

[2251] The system includes the following hardware and software:

[2252] Smartphones or smart glasses: Equipped with a camera and microphone, they collect user interaction and emotion data.

[2253] EmotionRecognizer class: Analyzes user emotions using an open-source emotion recognition library.

[2254] TaskAutomator class: A custom library for automating tasks.

[2255] Server: Stores operation data and emotion data and performs data analysis using machine learning algorithms.

[2256] Program processing

[2257] The server first launches an application on a smartphone or smart glasses to collect user operation data in real time. The application records the employee's clicks, keystrokes, system startup and shutdown, etc. It also uses the built-in camera and microphone to collect emotional data from facial expressions and tone of voice.

[2258] The collected data is sent to a server at regular intervals, where it is stored in a database of both user operation and emotion data. The server then uses machine learning algorithms to identify repetitive tasks and analyzes the emotion data to identify tasks that cause stress to the user.

[2259] For high-stress, frequently repetitive tasks, the system uses a generative AI model to suggest optimal automation methods, which are then embodied as scripts or automation tools and deployed from a server to users' devices.

[2260] The system installs the proposed automation tool on the user's device and optimizes it based on the configuration file. During this process, the system checks the operation of the automation tool and makes adjustments as necessary.

[2261] Employees only need to perform automated tasks, significantly reducing the amount of manual work they need to do, which increases work efficiency and reduces stress levels.

[2262] Specific examples

[2263] For example, suppose there is an employee who handles accounting at a cash register in a physical store. In this case, the system operates as follows:

[2264] 1. Operation Record and Emotion Recognition

[2265] Record employees scanning items into the cash register system.

[2266] Using the camera in smart glasses or the microphone in a smartphone, it analyzes employees' facial expressions and voice to identify their emotional state.

[2267] 2. Data transmission and analysis

[2268] The collected operation data and emotion data are sent to a server.

[2269] The server stores and analyzes this data.

[2270] 3. Identifying repetitive tasks and suggesting automation

[2271] Identify that cash register operations are repetitive tasks.

[2272] Emotional data can be used to identify stress experienced while operating a cash register.

[2273] As an automation suggestion, we generate scripts to automate product scanning and accounting processes.

[2274] 4. Implementing automation tools

[2275] The scripts proposed by the server are deployed to the user's device and configured.

[2276] 5. Execute automation

[2277] Automated scripts handle cash register operations, leaving employees to simply verify the transaction.

[2278] In this way, the burden on employees can be reduced and work efficiency can be improved.

[2279] Prompt Sentence Examples

[2280] A sample prompt is as follows:

[2281] "Record cashier activity, analyze sentiment data, and suggest tasks that can be automated, such as scanning items or printing receipts."

[2282] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2283] Step 1:

[2284] When a user begins work on a device, an application installed on the smartphone or smart glasses automatically starts in the background. The application records user operation data in real time, including clicks, keystrokes, system startup / shutdown, and file operations. At the same time, it uses the built-in camera and microphone to collect emotional data from facial expressions and tone of voice. As a result, the input data becomes operation data and emotional data. The output data is temporarily stored on the device as separate datasets.

[2285] Step 2:

[2286] The device transmits the recorded operation data and emotion data to the server at regular intervals. The transmission frequency is adjusted based on network load and security considerations, for example, every 30 minutes. The input to this step is the operation data and emotion data recorded in step 1, and the output is the data transmitted to the server.

[2287] Step 3:

[2288] The operation data and emotion data received by the server are immediately stored in a database. The stored data includes a timestamp, operation details, system information of the operation target, and the user's emotional state. The input of this step is the operation data and emotion data sent from the terminal, and the output is storage in the database.

[2289] Step 4:

[2290] The server uses a machine learning algorithm to analyze the stored data and identify specific operation patterns. Repetitive operations and regularly repeated tasks are identified as repetitive tasks. At the same time, it analyzes emotional data to identify which tasks employees experience stress during. The input for this step is the operation data and emotional data stored in the database, and the output is the identified repetitive tasks and stressful tasks.

[2291] Step 5:

[2292] The server proposes automation methods based on the repetitive tasks and emotion data identified using the generative AI model. The generative AI model provides optimal methods for automating tasks, such as script generation and automation tool configuration. It also adjusts automation priorities according to the user's stress level. The inputs for this step are the repetitive tasks and stress data identified in step 4, and the output is specific automation method proposals.

[2293] Step 6:

[2294] The server instantiates the proposed automation measures and generates scripts and configuration files. The generated data is ready to be deployed to the user's device. The input to this step is the automation measures proposed in step 5, and the output is the generated scripts and configuration files.

[2295] Step 7:

[2296] The terminal receives the deployed scripts and automation tools from the server, installs and configures them on the terminal, and performs the necessary tests to verify that the t...

Claims

1. A means for recording user operations; means for analyzing the recorded operations and identifying repetitive tasks; a means for suggesting automation measures based on the identified repetitive tasks; means for implementing the proposed automation means on a user's terminal; A system including:

2. means for storing the recorded operation data; A means for learning operation patterns based on the stored data; The system of claim 1 further comprising:

3. 10. The system of claim 1, further comprising means for evaluating the frequency of execution of the identified tasks and determining automation priorities based on the frequency.

Images