system

Abstract

Provide a system. 【Solution means】 Means for monitoring the operations of business performers and collecting information; Means for identifying repetitive work from the collected information; Means for generating an automation proposal for the identified work and notifying the business performer; Means for constructing a processing procedure for automating the approved work after the business performer approves the proposal; Means for automatically generating a program as needed and performing verification; Means for executing the automated work and reporting the results to the business performer; Means for displaying the automation proposal through a visualization device; Means for executing the work automated with the permission of the business performer; A system including the above.

Description

Technical Field

[0004] , , , ,

[0005] , , , , , ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the workplace, many employees spend time on many repetitive and routine tasks in their daily work. Such routine tasks take important time and prevent concentration on more strategic and creative activities. Although there are current automation tools, many companies need skills and time for their introduction and cannot utilize them effectively. This invention aims to facilitate business automation and improve business efficiency without the user having special technical knowledge.

Means for Solving the Problems

[0005] This invention provides a means for monitoring users' work activities, analyzing collected data, and identifying repetitive tasks. Furthermore, it generates automation suggestions for the identified tasks and notifies the user. These suggestions are user-approvable, and after approval, the system automatically builds a workflow to execute the tasks. It also includes features for automatically generating and debugging code as needed, enabling flexible automation even without technical knowledge. This allows users to focus on strategic tasks.

[0006] "Monitoring user work activities" means recording and collecting data on the daily work operations and actions that users perform.

[0007] "Means of data collection" refers to devices or processes that acquire necessary information from users' business activities and store it in a database or similar.

[0008] "Means for identifying repetitive tasks" refer to algorithms and methods for identifying tasks that are repeated at a certain frequency from collected data.

[0009] "Means for generating automation suggestions and notifying users" refers to technology that develops automation plans for efficiency improvements based on identified repetitive tasks and presents them to users.

[0010] "A means of building a workflow to automate approved tasks after a user has approved a proposal" refers to a system function that sets up automated processing procedures for tasks based on automation proposals approved by the user.

[0011] "Methods for automatically generating and debugging code" refer to techniques for creating programs to automate specific tasks and then verifying and correcting their accuracy and operation.

[0012] "A means of performing automated tasks and reporting the results to the user" refers to the process of executing a configured automated workflow and communicating the results to the user.

[0013] "Means that enable customization of automated suggestions based on user input" refers to a function that allows users to modify and specify the content of automated suggestions.

[0014] "Means of collecting user feedback and using it to improve the system" refers to technologies for gathering user opinions and evaluations after use and utilizing them to improve the system's functionality. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention provides an automation system for improving the work efficiency of users. This system aims to monitor users' work activities and automate repetitive tasks. Specific embodiments are described below.

[0037] Server Processing

[0038] The server establishes connections with the business tools that users use on a daily basis and monitors user activities. Examples include sending emails, editing spreadsheets, and managing tasks in project management tools. The server analyzes the data collected from these activities to identify repetitive tasks. For identified tasks, the server generates suggestions for automation. These suggestions include an overview of the efficiency gains from automation and the necessary configurations.

[0039] Terminal processing

[0040] The terminal displays automation proposals sent from the server to the user. The proposals include specific details of the automation, expected results, and any necessary customization information. When the user approves the proposal, the terminal sends that information back to the server, instructing it to start the automation process.

[0041] User processing

[0042] Users review automation suggestions they receive during their daily work. They consider the suggestions and approve them if they deem them useful. They can also customize the suggestions as needed. For example, they can adjust the target data range or change the execution timing to better fit their work.

[0043] As a concrete example, suppose a user is responsible for creating weekly team meeting minutes and sending them to everyone. The server identifies this task and generates a proposal to "automatically create meeting minutes from a template every Friday and email them to team members." Once the user approves this proposal, the server sets up the automated process and executes it automatically at the specified time. In this way, the user is freed from repetitive tasks and can focus on other important work.

[0044] The system of the present invention thus improves overall productivity by reducing the user's operational burden and optimizing business processes.

[0045] The following describes the processing flow.

[0046] Step 1:

[0047] The server establishes API connections with the user's business tools and monitors their daily work activities. This includes user email operations, spreadsheet editing, and task management in project management tools.

[0048] Step 2:

[0049] The server analyzes the collected business data and uses machine learning algorithms to identify repetitive tasks. In this process, pattern recognition techniques are used to identify operations that are repeated at a certain frequency.

[0050] Step 3:

[0051] Based on the identified tasks, the server evaluates the benefits of automation and generates automation proposals. These proposals detail areas where improvements in operational efficiency and time savings are expected.

[0052] Step 4:

[0053] The terminal displays automation suggestions sent from the server to the user on its interface. These suggestions include the tasks to be performed, the schedule, and customization options.

[0054] Step 5:

[0055] Users review the proposals via their terminal and provide input for approval or customization. Users can specify execution timing and conditions as needed.

[0056] Step 6:

[0057] The server builds automated workflows to fit specific business processes based on user-approved suggestions. At this stage, the necessary scripts and settings are automatically generated in the backend.

[0058] Step 7:

[0059] The server debugs the automatically generated code, detecting and correcting errors. This ensures the accuracy and stability of the code.

[0060] Step 8:

[0061] The server automatically executes the configured workflow according to a pre-configured schedule. The results of the execution are recorded and saved for evaluation.

[0062] Step 9:

[0063] The terminal notifies the user of the execution results. The success or failure status is displayed, allowing the user to re-execute or adjust the process as needed.

[0064] Step 10:

[0065] Users review the provided results and provide feedback as needed. This feedback is sent to the server and used to improve and optimize the system.

[0066] (Example 1)

[0067] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0068] In today's work environment, users often spend a significant amount of time on repetitive tasks, and there is a need to improve productivity. The challenge is to efficiently automate these repetitive tasks so that users can focus on more important work.

[0069] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0070] In this invention, the server includes means for monitoring the work activities of individual users using an information processing device and collecting operation logs, means for analyzing the collected operation logs to identify repetitive tasks, and means for creating automation suggestions for the identified tasks and communicating them to the user. This enables efficient execution of a process to automate repetitive tasks that users perform on a daily basis, thereby improving productivity.

[0071] An "information processing device" refers to a device that has the functions of inputting, processing, and outputting data, and is used to perform calculations and data manipulation.

[0072] "User" is a term that refers to an individual or group that performs specific operations using an information processing device, and is the ultimate beneficiary of the system.

[0073] "Work activity" refers to the specific operations or parts of a task that a user performs on an information processing device, and generally encompasses a series of actions required to complete a task.

[0074] An "operation log" refers to a collection of data recorded by an information processing device that shows the user's actions, and usually includes information that shows the user's behavior history.

[0075] "Analysis" refers to the process of using statistical or machine learning methods to elucidate the meaning and patterns of collected data.

[0076] "Repetitive tasks" refer to business processes in which the same operations are repeated over a certain period of time, and are usually targets for automation.

[0077] An "automation proposal" refers to a document that describes a series of procedures or methods devised to automate a specific operation, and is presented to the user.

[0078] A "processing procedure" refers to a set of sequential steps that a system follows to complete a specific task, and it forms the basis of automated operations.

[0079] "Program code" refers to a series of instructions written to cause an information processing device to perform a specific process.

[0080] "Means of fixing a problem" refers to methods for identifying the cause and implementing solutions when software or hardware does not function as intended.

[0081] "Execution result" refers to the output data generated as a result of instructions given by the user to the information processing device, and indicates whether the operation was successful or unsuccessful.

[0082] This invention is an automated system designed to improve the operational efficiency of users. The system is primarily implemented by three entities: the server, the terminal, and the user.

[0083] The server acts as an information processing device, establishing connections with business tools to effectively monitor user activity. Specifically, cloud-based email services, spreadsheet software, and task management applications are used. The server collects activity logs from these tools and analyzes them using machine learning algorithms. During the analysis, the server identifies repetitive tasks and generates suggestions for automation. This generation process utilizes cutting-edge artificial intelligence technology called generative AI models, which construct specific suggestions through prompt messages.

[0084] The terminal plays a crucial role in notifying users of automation proposals generated by the server. The proposal details, including the specific automation mechanism and expected deliverables, are displayed on the terminal's screen. Users can approve or modify the proposal via the terminal, and the results are then sent back to the server.

[0085] The user is the entity that actually performs the work and makes appropriate decisions based on the automation suggestions provided from the terminal. For example, if the user is responsible for creating meeting minutes after a regular weekly meeting and sending them to the relevant members, the server can suggest something like, "Automatically create and send meeting minutes every Friday using a template." This process is implemented using a prompt message like the following: "Please suggest a way for the user to automate the task of creating and sending meeting minutes after a regular weekly meeting."

[0086] By implementing this invention, users will be freed from repetitive tasks and will be able to concentrate on tasks that require higher-level judgment and creativity. The entire system is designed to reduce workload and improve efficiency through information processing.

[0087] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0088] Step 1:

[0089] The server establishes a connection with the user's business tools and monitors their work activities. Specifically, it accesses email clients and spreadsheet applications via APIs and obtains activity logs of the user's actions. Inputs include user activity logs, and output is log data stored in the server's database.

[0090] Step 2:

[0091] The server analyzes accumulated log data to identify recurring operations and tasks performed by users. Machine learning algorithms are used for the analysis, with log data provided as input. As output, the server identifies recurring work patterns and lists potential automation candidates based on them.

[0092] Step 3:

[0093] The server uses a generative AI model to generate automation proposals for the identified automation candidates. Prompt statements are used to generate proposals that describe specific automation steps and expected benefits. The input here is the identified automation candidates, and the output includes the generated proposals.

[0094] Step 4:

[0095] The terminal displays automation suggestions sent from the server to the user. The terminal screen presents the automation details and expected results for the user to review. The input is the automation suggestion sent from the server, and the output is a visual representation of the information conveyed to the user.

[0096] Step 5:

[0097] Users review the displayed proposals, modify the plan as needed, and finally approve it. The specific input is the proposal information displayed on the terminal, and the output is the approved or modified information sent to the server. Users operate this process using the terminal's interface.

[0098] Step 6:

[0099] The server receives approval or correction information from users and constructs the automated workflow. If necessary, it automatically generates program code and verifies that there are no operational problems. The input is user approval information, and the output is the configured automated process.

[0100] Step 7:

[0101] The server executes automated processes based on a schedule. For example, it can automatically send emails or generate documents at scheduled times. The input to the execution is the configured workflow, and the output is the completion status of the predetermined tasks reported to the user as a result of the execution.

[0102] (Application Example 1)

[0103] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0104] In modern operations, particularly in logistics centers and other work environments, there are many repetitive and time-consuming tasks. As a result, staff members spend a significant amount of time and effort on these tasks, hindering efficient operations. In this context, there is a need for effective systems that automate tasks and improve operational efficiency.

[0105] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0106] In this invention, the server includes means for monitoring the actions of the work performer and collecting information, means for identifying repetitive tasks from the collected information, and means for generating automation suggestions for the identified tasks and notifying the work performer. This enables the efficiency and automation of repetitive tasks in the workplace.

[0107] A "work performer" is the person who directly carries out the work and is the entity that makes decisions and performs operations related to the content of the work.

[0108] "Action" refers to a series of actions or work processes performed by a person carrying out a task.

[0109] "Information" refers to data and records obtained from the actions of those performing the work, and is used for analyzing the content of the work.

[0110] "Repetitive tasks" refer to work activities in which the same or similar actions are repeated regularly and continuously.

[0111] An "automation proposal" presents the person performing the work with specific methods and processes to streamline identified repetitive tasks.

[0112] "Notification" refers to the act of communicating automation suggestions or other information to those performing the work.

[0113] This invention provides a system for streamlining repetitive tasks in business sites such as logistics centers. This system monitors the actions of workers, identifies repetitive tasks from the information obtained, and generates and notifies users of suggestions for automation of those tasks.

[0114] The server monitors the actions of the work performers in real time and processes the collected information using data analysis software. This data analysis software incorporates machine learning algorithms, enabling high-precision identification of repetitive tasks. Based on the identified tasks, specific suggestions for automation are generated and communicated to the work performers. Visualization devices such as smart glasses and tablets are used in this process.

[0115] The person performing the task reviews the proposal displayed on the visualization device, customizes it as needed, and approves it. The approved proposal is sent to the server, which automatically generates a program and automates the work. This process allows the person performing the task to proceed more efficiently.

[0116] For example, if a logistics center staff member picks items from shelves along a similar route at a fixed time each day, this task can be identified, generating a proposal to "automate picking on this route at 10:00 AM every day." Once the staff member approves this proposal, a picking robot can perform the work along the automated route. This allows the staff member to focus on other important tasks.

[0117] An example of a prompt for a generating AI model is: "Generate prompts for the system to recognize the user's daily work events and identify parts that can be automated. For example, if the same task is repeated at a specific time, consider suggestions for automating it." This prompt provides the AI ​​model with the information necessary to generate automation suggestions.

[0118] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0119] Step 1:

[0120] The server monitors the actions of the worker through smart glasses or cameras and collects the video data. The input is real-time video data, and the output is the action information to be analyzed. In this step, the video data is temporarily stored in preparation for analysis in the next step.

[0121] Step 2:

[0122] The server processes the collected video data using data analysis software and identifies repetitive tasks using machine learning algorithms. The input is the behavioral information obtained in step 1, and the output is a list of identified repetitive tasks. Data analysis involves time-series data analysis and clustering to find specific patterns.

[0123] Step 3:

[0124] The server generates automation suggestions based on identified repetitive tasks. The input is the list of repetitive tasks obtained from step 2, and the output is the automation suggestions. A generative AI model is used for generation, and automation proposals are formulated based on past successes and criteria for operational efficiency.

[0125] Step 4:

[0126] The terminal notifies the task implementer of the generated automation proposals and displays them on a visualization device. The input is the automation proposals sent from the server, and the output is the notification to the task implementer. This operation allows the implementer to understand the content of the proposals.

[0127] Step 5:

[0128] The user reviews the automation proposal displayed on the terminal and performs customization or approval operations as needed. The input is the automation proposal from step 4, and the output is the approved proposal or customized content. Through this operation, the implementer makes adjustments according to specific work conditions.

[0129] Step 6:

[0130] The server receives approved proposals from users and automatically generates programs to prepare for automated task execution. The input is user approval information, and the output is automatically generated program code. A code generation tool is used for program generation, and debugging is performed concurrently during the generation process.

[0131] Step 7:

[0132] The server executes automatically generated programs to automate repetitive tasks. The input is the program code from step 6, and the output is the completed result of the automated work. Clear time management and resource allocation are implemented during this execution.

[0133] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0134] This invention combines an automated system for improving user work efficiency with an emotion engine that recognizes user emotions. The system aims not only to monitor user work activities and automate repetitive tasks, but also to generate suggestions that take the user's emotional state into account. Specific embodiments are described below.

[0135] Server Processing

[0136] The server monitors work activities through the business tools used by the user. This includes user interactions in email, spreadsheets, and project management applications. The server collects this activity data and uses machine learning algorithms to identify repetitive tasks.

[0137] Furthermore, the server utilizes an emotion engine to evaluate the user's emotions by analyzing non-verbal data such as the user's voice, facial expressions, and typing speed. This emotion data is then used to suggest automation improvements for increased work efficiency. For example, if a user is experiencing stress, the system will lower the priority of that task and generate suggestions to automate other tasks instead.

[0138] Terminal processing

[0139] The terminal displays automated suggestions sent from the server to the user. These suggestions include adjustments based on analysis by the emotion engine. For example, to reduce user stress, suggestions may be made to distribute the workload.

[0140] Users can review these proposals from their devices and approve or customize them. Approved proposals are sent to the server, and the automated workflow is built.

[0141] User processing

[0142] Users can receive automation suggestions and sentiment-based analysis results in real time, allowing them to adjust their work processes. For example, if a particular task is determined to be emotionally demanding, the user can reschedule that task.

[0143] As a concrete example, suppose a user repeatedly performs the task of creating daily reports. The server identifies this task as one that can be automated and also determines that the user is under stress based on their facial expression and slow input speed. In this case, the system reduces the user's burden by suggesting that "data collection for the daily reports be automated, and the user only edits the parts they need to edit."

[0144] Thus, the system of the present invention can optimize operations by taking into account not only the user's workload but also their psychological state, thereby providing a more effective work environment.

[0145] The following describes the processing flow.

[0146] Step 1:

[0147] The server integrates with users' business tools to monitor their daily work activities. Specifically, it acquires operation logs from email software, spreadsheets, project management tools, and other similar applications.

[0148] Step 2:

[0149] The server uses an emotion engine to collect nonverbal data in real time, such as the user's voice tone, facial expressions, and typing speed. This allows the server to evaluate the user's emotional state.

[0150] Step 3:

[0151] The server analyzes business data and uses machine learning algorithms to identify repetitive tasks. For example, this includes tasks such as writing reports that are performed at the same time every day.

[0152] Step 4:

[0153] Based on the analysis results from the emotion engine, the server adjusts task priorities and generates optimal automation suggestions for the user. Specifically, if the user is experiencing stress, suggestions will be made to reduce the workload of that task.

[0154] Step 5:

[0155] The terminal displays automated suggestions received from the server to the user via its interface. At this stage, the suggestions incorporate adjustments based on sentiment analysis.

[0156] Step 6:

[0157] Users can review the proposals via their device and approve or customize them. They can also set execution timings and specific conditions as needed.

[0158] Step 7:

[0159] The server builds a customized automated workflow based on the user's approved proposal, automatically creating the necessary scripts and task configurations.

[0160] Step 8:

[0161] The server executes the configured workflow according to the set schedule. After execution, it records the results and saves them to the database.

[0162] Step 9:

[0163] The terminal notifies the user of the execution results. This includes success, failure, and feedback collection functions as needed.

[0164] Step 10:

[0165] Users review the execution results and provide feedback. This information is sent to the server and used to improve the emotion engine and automation system.

[0166] (Example 2)

[0167] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0168] In today's business environment, many tasks performed by users involve repetitive work, which contributes to decreased work efficiency. Furthermore, stress and emotional burden can directly impact work performance. However, conventional automation systems have difficulty flexibly responding to users' emotional states, making it challenging to alleviate their psychological burden.

[0169] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0170] In this invention, the server includes means for monitoring the user's work activities via a processing device and collecting information; means for identifying repetitive tasks from the collected information and performing analysis using a machine learning algorithm; and means for analyzing the user's nonverbal information and adjusting the suggested content by evaluating their emotional state. This enables automated suggestions that take into account the user's emotional state, improving work efficiency and reducing emotional burden.

[0171] "Users" refers to individuals or groups who actually use the system and conduct business activities using it.

[0172] "Business activities" refers to the collective term for the operations and tasks performed by users in the course of their jobs.

[0173] A "processing device" refers to a mechanical or electronic device used to collect, analyze, store, or output data.

[0174] "Information" refers to content that the system acquires or generates for processing and analysis, such as users' work activities and non-verbal data.

[0175] "Repetitive tasks" refer to work tasks that involve the same procedures or processes occurring repeatedly.

[0176] A "machine learning algorithm" refers to a computational method that learns patterns from data and automatically performs analysis and prediction.

[0177] "Non-verbal information" refers to data other than linguistic information, such as the user's voice, facial expressions, and typing speed.

[0178] "Emotional state" refers to the user's psychological state, including stress levels, satisfaction levels, and motivation.

[0179] "Adjusting the proposed content" refers to modifying automated suggestions based on the user's emotional state and work situation.

[0180] This invention is implemented by a system consisting of a server, terminals, and users. The server first monitors business activities through business software used by the users. Specifically, the server uses a computing device equipped with a high-performance processor and large-capacity storage. The server collects data via APIs of software used by the users, such as email, spreadsheets, and project management software, and stores it in a database.

[0181] The server then runs machine learning algorithms using the collected data. This execution uses the Python language and the scikit-learn library, and includes cluster analysis to identify repetitive tasks.

[0182] Furthermore, the server uses an emotion engine to analyze non-verbal data such as the user's voice, facial expressions, and input speed. This utilizes speech recognition software and facial expression analysis libraries (e.g., OpenCV).

[0183] The data generated from the analysis will be used to generate automation suggestions in the form of prompts using a generation AI model (e.g., GPT-3®). The aim of these suggestions is to reduce the user's workload and to make adjustments that take emotional states into consideration.

[0184] The terminal's role is to visually display automation suggestions received from the server to the user. This uses HTML and JavaScript (registered trademark) to build the user interface.

[0185] Users can review these proposals via their devices and approve or adjust them. Approved proposals are sent back to the server, and the necessary workflows are automated.

[0186] To give a concrete example, in a user's daily report-writing task, the server suggests automating the data entry portion. This allows the user to focus on reviewing and editing the necessary parts of the report, saving time and effort.

[0187] An example of a prompt message is generated in the format, "To what extent will automating this task reduce user stress?"

[0188] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0189] Step 1:

[0190] The server collects user work activity data through the APIs of business software. Inputs include emails, spreadsheets, and project management tool operation data. This data is stored in a database, making it available for subsequent analysis processes. Specifically, the system is designed to periodically call the APIs to retrieve the latest work activity information.

[0191] Step 2:

[0192] The server executes machine learning algorithms using the collected business data. This step uses business activity data stored in a database as input. Clustering analysis is performed using the scikit-learn library to identify repetitive tasks. The output is a list of identified repetitive tasks. Real-time functionality is achieved by running a Python script as a scheduled job.

[0193] Step 3:

[0194] The server uses an emotion engine to analyze nonverbal data and evaluate the user's emotional state. Inputs include voice, facial expression data, and input speed. These data are analyzed using speech recognition software and OpenCV to obtain numerical data for each emotional state. The output is an evaluation result indicating the user's emotional state. Specifically, the server is configured to process video and audio data in real time.

[0195] Step 4:

[0196] The server generates automation suggestions using a generative AI model. This step uses identified repetitive tasks and emotional state evaluation results as input data. The generative AI model receives prompt text and outputs the optimal automation suggestion. Specifically, the generative AI model is provided on a cloud service, and suggestion generation is performed via an API.

[0197] Step 5:

[0198] The terminal displays automation suggestions received from the server to the user. It receives suggestion data from the server as input and displays the suggested content on the user interface as output. Specifically, it uses HTML and JavaScript to build a visually appealing interface, ensuring that the suggestions are intuitively understandable to the user.

[0199] Step 6:

[0200] Users review proposals via a terminal and approve or adjust them as needed. Input involves reviewing the proposal displayed on the terminal's interface and entering their own opinions and requests. Output is the adjusted proposal information, which is sent to the server. Specific actions involve fine-tuning the proposal using a mouse and keyboard.

[0201] Step 7:

[0202] The server prepares to execute automated workflows based on user approvals. It receives adjusted proposal information as input and executes automation scripts and tasks as output. Specifically, automated task processing using RPA tools is scheduled on the server.

[0203] (Application Example 2)

[0204] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0205] Conventional business automation systems automate tasks without considering the user's emotional state, making it difficult to reduce the user's psychological burden. Repetitive administrative tasks, in particular, can increase stress, so task adjustments that take emotional states into account are needed.

[0206] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0207] In this invention, the server includes means for monitoring the user's work activities and collecting data; means for identifying repetitive administrative tasks from the collected data; means for generating automation suggestions for the identified administrative tasks and notifying the user; means for providing feedback on the emotional state via a display device as needed; means for analyzing the user's facial expressions and voice and evaluating their emotions; and means for making suggestions for improving work efficiency based on the emotional evaluation. This enables efficient work execution that takes into account the user's emotional state.

[0208] A "user" is the entity that utilizes the system and is the target of suggestions for automating repetitive administrative tasks and improving efficiency based on emotional evaluations through their work activities.

[0209] "Business activities" refer to the duties and tasks that users perform on a daily basis, and the actions that are subject to monitoring and data collection.

[0210] "Data" includes users' work activities, facial expressions, voice, and information derived therefrom, and is collected for the purpose of identifying repetitive clerical tasks and evaluating emotions.

[0211] "Office work" refers to tasks that are particularly routine and repetitive within business activities, and are therefore subject to automation.

[0212] "Automation suggestions" refer to system recommendations for improving the efficiency of identified administrative tasks.

[0213] A "display device" refers to hardware used to present information to a user, either visually or audibly.

[0214] "Emotional state" refers to the user's psychological state and is evaluated based on information such as facial expressions and voice.

[0215] "Emotional evaluation" refers to the process of analyzing a user's facial expressions and voice information to determine their psychological state.

[0216] "Efficiency suggestions" refer to specific recommendations made based on emotional evaluations to improve the user's work efficiency.

[0217] In embodiments of this invention, it is important to design a system including a server and a display device to support the user's work activities. First, the server monitors the user's work activities and collects data. This data includes operations on email, spreadsheets, project management applications, etc., that the user uses on a daily basis. The server uses this data to identify repetitive administrative tasks.

[0218] Furthermore, the server utilizes an emotion engine to evaluate the user's emotional state from their facial expressions and voice. Specifically, it employs speech recognition software and image analysis technology to analyze the user's voice tone and facial expressions in real time. For example, it uses Python and the OpenCV library, and the Hugging Face Transformers library for voice analysis. This allows the system to understand the user's emotional state and generate optimal suggestions for improving work efficiency based on that understanding.

[0219] If a user's stress level is high, the server can lower the priority of that task and suggest automating other, easier tasks. The generated suggestions are notified to the user via a display device, and the user can approve or adjust the suggestions.

[0220] For example, if a user's regular report-writing task is identified as a source of stress, the server might suggest "automating data collection for reports and only requiring annotations." Such suggestions help users perform their tasks efficiently with less emotional burden. An example of a prompt might be, "If a user is experiencing stress, what work adjustment suggestions would you offer?"

[0221] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0222] Step 1:

[0223] The server monitors user work activities and collects operational data such as emails and spreadsheets. This collected data is used later to identify repetitive administrative tasks. Inputs include real-time user operational data obtained from business software, and outputs are organized data streams.

[0224] Step 2:

[0225] The server analyzes the collected data and uses machine learning models to identify repetitive tasks. This identification process determines whether a particular task is repeated by recognizing patterns in the data. The input is the collected data stream, and the output is a list of identified tasks.

[0226] Step 3:

[0227] The server uses an emotion engine to analyze the user's facial expressions and voice to evaluate their emotional state. It utilizes speech recognition and image analysis technologies to assess how emotions are changing in real time. Specifically, it takes camera video and audio data as input and outputs emotion data as the analysis result.

[0228] Step 4:

[0229] The server generates automation suggestions for improving work efficiency based on identified administrative tasks and user sentiment data. A generative AI model, trained on historical data, is used to generate these suggestions. Inputs include a list of administrative tasks and sentiment data, while output is optimized automation suggestions.

[0230] Step 5:

[0231] The terminal displays automated suggestions sent from the server, allowing the user to approve and adjust the suggestions. The user can review these suggestions and customize them to their preferences. Suggestions are input from the display device, and the user's selections are recorded as the output of the operation.

[0232] Step 6:

[0233] If a user approves a proposal, the server automates the process based on the approved administrative task. The input includes the user-approved proposal, and the output is the automated workflow deployed within the system.

[0234] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0235] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0236] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0237] [Second Embodiment]

[0238] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0239] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0240] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0241] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0242] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0243] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0244] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0245] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0246] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0247] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0248] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0249] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0250] This invention provides an automation system for improving the work efficiency of users. This system aims to monitor users' work activities and automate repetitive tasks. Specific embodiments are described below.

[0251] Server Processing

[0252] The server establishes connections with the business tools that users use on a daily basis and monitors user activities. Examples include sending emails, editing spreadsheets, and managing tasks in project management tools. The server analyzes the data collected from these activities to identify repetitive tasks. For identified tasks, the server generates suggestions for automation. These suggestions include an overview of the efficiency gains from automation and the necessary configurations.

[0253] Terminal processing

[0254] The terminal displays automation proposals sent from the server to the user. The proposals include specific details of the automation, expected results, and any necessary customization information. When the user approves the proposal, the terminal sends that information back to the server, instructing it to start the automation process.

[0255] User processing

[0256] Users review automation suggestions they receive during their daily work. They consider the suggestions and approve them if they deem them useful. They can also customize the suggestions as needed. For example, they can adjust the target data range or change the execution timing to better fit their work.

[0257] As a concrete example, suppose a user is responsible for creating weekly team meeting minutes and sending them to everyone. The server identifies this task and generates a proposal to "automatically create meeting minutes from a template every Friday and email them to team members." Once the user approves this proposal, the server sets up the automated process and executes it automatically at the specified time. In this way, the user is freed from repetitive tasks and can focus on other important work.

[0258] The system of the present invention thus improves overall productivity by reducing the user's operational burden and optimizing business processes.

[0259] The following describes the processing flow.

[0260] Step 1:

[0261] The server establishes API connections with the user's business tools and monitors their daily work activities. This includes user email operations, spreadsheet editing, and task management in project management tools.

[0262] Step 2:

[0263] The server analyzes the collected business data and uses machine learning algorithms to identify repetitive tasks. In this process, pattern recognition techniques are used to identify operations that are repeated at a certain frequency.

[0264] Step 3:

[0265] Based on the identified tasks, the server evaluates the benefits of automation and generates automation proposals. These proposals detail areas where improvements in operational efficiency and time savings are expected.

[0266] Step 4:

[0267] The terminal displays automation suggestions sent from the server to the user on its interface. These suggestions include the tasks to be performed, the schedule, and customization options.

[0268] Step 5:

[0269] Users review the proposals via their terminal and provide input for approval or customization. Users can specify execution timing and conditions as needed.

[0270] Step 6:

[0271] The server builds automated workflows to fit specific business processes based on user-approved suggestions. At this stage, the necessary scripts and settings are automatically generated in the backend.

[0272] Step 7:

[0273] The server debugs the automatically generated code, detecting and correcting errors. This ensures the accuracy and stability of the code.

[0274] Step 8:

[0275] The server automatically executes the configured workflow according to a pre-configured schedule. The results of the execution are recorded and saved for evaluation.

[0276] Step 9:

[0277] The terminal notifies the user of the execution results. The success or failure status is displayed, allowing the user to re-execute or adjust the process as needed.

[0278] Step 10:

[0279] Users review the provided results and provide feedback as needed. This feedback is sent to the server and used to improve and optimize the system.

[0280] (Example 1)

[0281] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0282] In a modern business environment, users often spend a lot of time on repetitive tasks, and there is a need to improve productivity. The challenge is to efficiently automate such repetitive tasks so that users can focus on more important tasks.

[0283] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0284] In this invention, the server includes means for monitoring the work activities of individual users by an information processing device and collecting operation logs, means for analyzing the collected operation logs to identify repetitive tasks, and means for creating and communicating proposals for automation to the users for the identified tasks. This enables the efficient execution of the process of automating the repetitive tasks that users perform daily, thereby improving productivity.

[0285] The "information processing device" refers to a device having the function of inputting, processing, and outputting data, and is for performing calculations and data operations.

[0286] The "user" is a term referring to an individual or group that performs specific operations using an information processing device and is the ultimate beneficiary of the system.

[0287] The "work activity" refers to specific operations or parts of work that a user performs on an information processing device, and generally is a series of actions for completing a task.

[0288] The "operation log" refers to a collection of data recorded by an information processing device of user operations and usually includes information indicating the user's action history.

[0289] "Analysis" refers to a process using statistical or machine learning methods to clarify the meaning and patterns of the collected data.

[0290] "Repetitive tasks" refer to business processes in which the same operations are repeated over a certain period of time, and are usually targets for automation.

[0291] An "automation proposal" refers to a document that describes a series of procedures or methods devised to automate a specific operation, and is presented to the user.

[0292] A "processing procedure" refers to a set of sequential steps that a system follows to complete a specific task, and it forms the basis of automated operations.

[0293] "Program code" refers to a series of instructions written to cause an information processing device to perform a specific process.

[0294] "Means of fixing a problem" refers to methods for identifying the cause and implementing solutions when software or hardware does not function as intended.

[0295] "Execution result" refers to the output data generated as a result of instructions given by the user to the information processing device, and indicates whether the operation was successful or unsuccessful.

[0296] This invention is an automated system designed to improve the operational efficiency of users. The system is primarily implemented by three entities: the server, the terminal, and the user.

[0297] The server acts as an information processing device, establishing connections with business tools to effectively monitor user activity. Specifically, cloud-based email services, spreadsheet software, and task management applications are used. The server collects activity logs from these tools and analyzes them using machine learning algorithms. During the analysis, the server identifies repetitive tasks and generates suggestions for automation. This generation process utilizes cutting-edge artificial intelligence technology called generative AI models, which construct specific suggestions through prompt messages.

[0298] The terminal plays a crucial role in notifying users of automation proposals generated by the server. The proposal details, including the specific automation mechanism and expected deliverables, are displayed on the terminal's screen. Users can approve or modify the proposal via the terminal, and the results are then sent back to the server.

[0299] The user is the entity that actually performs the work and makes appropriate decisions based on the automation suggestions provided from the terminal. For example, if the user is responsible for creating meeting minutes after a regular weekly meeting and sending them to the relevant members, the server can suggest something like, "Automatically create and send meeting minutes every Friday using a template." This process is implemented using a prompt message like the following: "Please suggest a way for the user to automate the task of creating and sending meeting minutes after a regular weekly meeting."

[0300] By implementing this invention, users will be freed from repetitive tasks and will be able to concentrate on tasks that require higher-level judgment and creativity. The entire system is designed to reduce workload and improve efficiency through information processing.

[0301] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0302] Step 1:

[0303] The server establishes a connection with the user's business tools and monitors their work activities. Specifically, it accesses email clients and spreadsheet applications via APIs and obtains activity logs of the user's actions. Inputs include user activity logs, and output is log data stored in the server's database.

[0304] Step 2:

[0305] The server analyzes the accumulated log data to identify the operations and tasks that the user repeats. Machine learning algorithms are used for the analysis, and log data is provided as the input. As the output, the server identifies the repetitive business patterns and lists the potential automation candidates based on them.

[0306] Step 3:

[0307] The server uses the generative AI model to generate automation proposals for the identified automation candidates. Proposals that describe the specific automation procedures and expected benefits are generated using prompt sentences. The input here is the identified automation candidates, and the output includes the generated proposals.

[0308] Step 4:

[0309] The terminal displays the automation proposals sent from the server to the user. The details of the automation and the expected results are presented on the terminal screen, making it possible for the user to confirm. The input is the automation proposal sent from the server, and the output is the visual transmission of information to the user.

[0310] Step 5:

[0311] The user examines the displayed proposal, modifies the plan if necessary, and finally gives approval. The specific input is the proposal information displayed on the terminal, and the output is the approved or modified information sent to the server. The user operates this using the terminal interface.

[0312] Step 6:

[0313] The server receives the approval or modification information from the user and specifically constructs the automation workflow. If necessary, it automatically generates program code and verifies whether there are any operational problems. The input is the user's approval information, and the output is the configured automation process.

[0314] Step 7:

[0315] The server executes automated processes based on a schedule. For example, it can automatically send emails or generate documents at scheduled times. The input to the execution is the configured workflow, and the output is the completion status of the predetermined tasks reported to the user as a result of the execution.

[0316] (Application Example 1)

[0317] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0318] In modern operations, particularly in logistics centers and other work environments, there are many repetitive and time-consuming tasks. As a result, staff members spend a significant amount of time and effort on these tasks, hindering efficient operations. In this context, there is a need for effective systems that automate tasks and improve operational efficiency.

[0319] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0320] In this invention, the server includes means for monitoring the actions of the work performer and collecting information, means for identifying repetitive tasks from the collected information, and means for generating automation suggestions for the identified tasks and notifying the work performer. This enables the efficiency and automation of repetitive tasks in the workplace.

[0321] A "work performer" is the person who directly carries out the work and is the entity that makes decisions and performs operations related to the content of the work.

[0322] "Action" refers to a series of actions or work processes performed by a person carrying out a task.

[0323] "Information" refers to data and records obtained from the actions of those performing the work, and is used for analyzing the content of the work.

[0324] "Repetitive tasks" refer to work activities in which the same or similar actions are repeated regularly and continuously.

[0325] An "automation proposal" presents the person performing the work with specific methods and processes to streamline identified repetitive tasks.

[0326] "Notification" refers to the act of communicating automation suggestions or other information to those performing the work.

[0327] This invention provides a system for streamlining repetitive tasks in business sites such as logistics centers. This system monitors the actions of workers, identifies repetitive tasks from the information obtained, and generates and notifies users of suggestions for automation of those tasks.

[0328] The server monitors the actions of the work performers in real time and processes the collected information using data analysis software. This data analysis software incorporates machine learning algorithms, enabling high-precision identification of repetitive tasks. Based on the identified tasks, specific suggestions for automation are generated and communicated to the work performers. Visualization devices such as smart glasses and tablets are used in this process.

[0329] The person performing the task reviews the proposal displayed on the visualization device, customizes it as needed, and approves it. The approved proposal is sent to the server, which automatically generates a program and automates the work. This process allows the person performing the task to proceed more efficiently.

[0330] For example, if a logistics center staff member picks items from shelves along a similar route at a fixed time each day, this task can be identified, generating a proposal to "automate picking on this route at 10:00 AM every day." Once the staff member approves this proposal, a picking robot can perform the work along the automated route. This allows the staff member to focus on other important tasks.

[0331] An example of a prompt for a generating AI model is: "Generate prompts for the system to recognize the user's daily work events and identify parts that can be automated. For example, if the same task is repeated at a specific time, consider suggestions for automating it." This prompt provides the AI ​​model with the information necessary to generate automation suggestions.

[0332] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0333] Step 1:

[0334] The server monitors the actions of the worker through smart glasses or cameras and collects the video data. The input is real-time video data, and the output is the action information to be analyzed. In this step, the video data is temporarily stored in preparation for analysis in the next step.

[0335] Step 2:

[0336] The server processes the collected video data using data analysis software and identifies repetitive tasks using machine learning algorithms. The input is the behavioral information obtained in step 1, and the output is a list of identified repetitive tasks. Data analysis involves time-series data analysis and clustering to find specific patterns.

[0337] Step 3:

[0338] The server generates automation suggestions based on identified repetitive tasks. The input is the list of repetitive tasks obtained from step 2, and the output is the automation suggestions. A generative AI model is used for generation, and automation proposals are formulated based on past successes and criteria for operational efficiency.

[0339] Step 4:

[0340] The terminal notifies the task implementer of the generated automation proposals and displays them on a visualization device. The input is the automation proposals sent from the server, and the output is the notification to the task implementer. This operation allows the implementer to understand the content of the proposals.

[0341] Step 5:

[0342] The user reviews the automation proposal displayed on the terminal and performs customization or approval operations as needed. The input is the automation proposal from step 4, and the output is the approved proposal or customized content. Through this operation, the implementer makes adjustments according to specific work conditions.

[0343] Step 6:

[0344] The server receives approved proposals from users and automatically generates programs to prepare for automated task execution. The input is user approval information, and the output is automatically generated program code. A code generation tool is used for program generation, and debugging is performed concurrently during the generation process.

[0345] Step 7:

[0346] The server executes automatically generated programs to automate repetitive tasks. The input is the program code from step 6, and the output is the completed result of the automated work. Clear time management and resource allocation are implemented during this execution.

[0347] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0348] This invention combines an automated system for improving user work efficiency with an emotion engine that recognizes user emotions. The system aims not only to monitor user work activities and automate repetitive tasks, but also to generate suggestions that take the user's emotional state into account. Specific embodiments are described below.

[0349] Server Processing

[0350] The server monitors work activities through the business tools used by the user. This includes user interactions in email, spreadsheets, and project management applications. The server collects this activity data and uses machine learning algorithms to identify repetitive tasks.

[0351] Furthermore, the server utilizes an emotion engine to evaluate the user's emotions by analyzing non-verbal data such as the user's voice, facial expressions, and typing speed. This emotion data is then used to suggest automation improvements for increased work efficiency. For example, if a user is experiencing stress, the system will lower the priority of that task and generate suggestions to automate other tasks instead.

[0352] Terminal processing

[0353] The terminal displays automated suggestions sent from the server to the user. These suggestions include adjustments based on analysis by the emotion engine. For example, to reduce user stress, suggestions may be made to distribute the workload.

[0354] Users can review these proposals from their devices and approve or customize them. Approved proposals are sent to the server, and the automated workflow is built.

[0355] User processing

[0356] Users can receive automation suggestions and sentiment-based analysis results in real time, allowing them to adjust their work processes. For example, if a particular task is determined to be emotionally demanding, the user can reschedule that task.

[0357] As a concrete example, suppose a user repeatedly performs the task of creating daily reports. The server identifies this task as one that can be automated and also determines that the user is under stress based on their facial expression and slow input speed. In this case, the system reduces the user's burden by suggesting that "data collection for the daily reports be automated, and the user only edits the parts they need to edit."

[0358] Thus, the system of the present invention can optimize operations by taking into account not only the user's workload but also their psychological state, thereby providing a more effective work environment.

[0359] The following describes the processing flow.

[0360] Step 1:

[0361] The server integrates with users' business tools to monitor their daily work activities. Specifically, it acquires operation logs from email software, spreadsheets, project management tools, and other similar applications.

[0362] Step 2:

[0363] The server uses an emotion engine to collect nonverbal data in real time, such as the user's voice tone, facial expressions, and typing speed. This allows the server to evaluate the user's emotional state.

[0364] Step 3:

[0365] The server analyzes business data and uses machine learning algorithms to identify repetitive tasks. For example, this includes tasks such as writing reports that are performed at the same time every day.

[0366] Step 4:

[0367] Based on the analysis results from the emotion engine, the server adjusts task priorities and generates optimal automation suggestions for the user. Specifically, if the user is experiencing stress, suggestions will be made to reduce the workload of that task.

[0368] Step 5:

[0369] The terminal displays automated suggestions received from the server to the user via its interface. At this stage, the suggestions incorporate adjustments based on sentiment analysis.

[0370] Step 6:

[0371] Users can review the proposals via their device and approve or customize them. They can also set execution timings and specific conditions as needed.

[0372] Step 7:

[0373] The server builds a customized automated workflow based on the user's approved proposal, automatically creating the necessary scripts and task configurations.

[0374] Step 8:

[0375] The server executes the configured workflow according to the set schedule. After execution, it records the results and saves them to the database.

[0376] Step 9:

[0377] The terminal notifies the user of the execution results. This includes success, failure, and feedback collection functions as needed.

[0378] Step 10:

[0379] Users review the execution results and provide feedback. This information is sent to the server and used to improve the emotion engine and automation system.

[0380] (Example 2)

[0381] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0382] In today's business environment, many tasks performed by users involve repetitive work, which contributes to decreased work efficiency. Furthermore, stress and emotional burden can directly impact work performance. However, conventional automation systems have difficulty flexibly responding to users' emotional states, making it challenging to alleviate their psychological burden.

[0383] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0384] In this invention, the server includes means for monitoring the user's work activities via a processing device and collecting information; means for identifying repetitive tasks from the collected information and performing analysis using a machine learning algorithm; and means for analyzing the user's nonverbal information and adjusting the suggested content by evaluating their emotional state. This enables automated suggestions that take into account the user's emotional state, improving work efficiency and reducing emotional burden.

[0385] "Users" refers to individuals or groups who actually use the system and conduct business activities using it.

[0386] "Business activities" refers to the collective term for the operations and tasks performed by users in the course of their jobs.

[0387] A "processing device" refers to a mechanical or electronic device used to collect, analyze, store, or output data.

[0388] "Information" refers to content that the system acquires or generates for processing and analysis, such as users' work activities and non-verbal data.

[0389] "Repetitive tasks" refer to work tasks that involve the same procedures or processes occurring repeatedly.

[0390] A "machine learning algorithm" refers to a computational method that learns patterns from data and automatically performs analysis and prediction.

[0391] "Non-verbal information" refers to data other than linguistic information, such as the user's voice, facial expressions, and typing speed.

[0392] "Emotional state" refers to the user's psychological state, including stress levels, satisfaction levels, and motivation.

[0393] "Adjusting the proposed content" refers to modifying automated suggestions based on the user's emotional state and work situation.

[0394] This invention is implemented by a system consisting of a server, terminals, and users. The server first monitors business activities through business software used by the users. Specifically, the server uses a computing device equipped with a high-performance processor and large-capacity storage. The server collects data via APIs of software used by the users, such as email, spreadsheets, and project management software, and stores it in a database.

[0395] The server then runs machine learning algorithms using the collected data. This execution uses the Python language and the scikit-learn library, and includes cluster analysis to identify repetitive tasks.

[0396] Furthermore, the server uses an emotion engine to analyze non-verbal data such as the user's voice, facial expressions, and input speed. This utilizes speech recognition software and facial expression analysis libraries (e.g., OpenCV).

[0397] The data generated from the analysis will be used to generate automation suggestions in the form of prompts using a generative AI model (e.g., GPT-3). The aim of these suggestions is to reduce the user's workload and to make adjustments that take emotional states into consideration.

[0398] The terminal's role is to visually display automation suggestions received from the server to the user. This uses HTML and JavaScript to build the user interface.

[0399] Users can review these proposals via their devices and approve or adjust them. Approved proposals are sent back to the server, and the necessary workflows are automated.

[0400] To give a concrete example, in a user's daily report-writing task, the server suggests automating the data entry portion. This allows the user to focus on reviewing and editing the necessary parts of the report, saving time and effort.

[0401] An example of a prompt message is generated in the format, "To what extent will automating this task reduce user stress?"

[0402] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0403] Step 1:

[0404] The server collects user work activity data through the APIs of business software. Inputs include emails, spreadsheets, and project management tool operation data. This data is stored in a database, making it available for subsequent analysis processes. Specifically, the system is designed to periodically call the APIs to retrieve the latest work activity information.

[0405] Step 2:

[0406] The server executes machine learning algorithms using the collected business data. This step uses business activity data stored in a database as input. Clustering analysis is performed using the scikit-learn library to identify repetitive tasks. The output is a list of identified repetitive tasks. Real-time functionality is achieved by running a Python script as a scheduled job.

[0407] Step 3:

[0408] The server uses an emotion engine to analyze nonverbal data and evaluate the user's emotional state. Inputs include voice, facial expression data, and input speed. These data are analyzed using speech recognition software and OpenCV to obtain numerical data for each emotional state. The output is an evaluation result indicating the user's emotional state. Specifically, the server is configured to process video and audio data in real time.

[0409] Step 4:

[0410] The server generates automation suggestions using a generative AI model. This step uses identified repetitive tasks and emotional state evaluation results as input data. The generative AI model receives prompt text and outputs the optimal automation suggestion. Specifically, the generative AI model is provided on a cloud service, and suggestion generation is performed via an API.

[0411] Step 5:

[0412] The terminal displays automation suggestions received from the server to the user. It receives suggestion data from the server as input and displays the suggested content on the user interface as output. Specifically, it uses HTML and JavaScript to build a visually appealing interface, ensuring that the suggestions are intuitively understandable to the user.

[0413] Step 6:

[0414] Users review proposals via a terminal and approve or adjust them as needed. Input involves reviewing the proposal displayed on the terminal's interface and entering their own opinions and requests. Output is the adjusted proposal information, which is sent to the server. Specific actions involve fine-tuning the proposal using a mouse and keyboard.

[0415] Step 7:

[0416] The server prepares to execute automated workflows based on user approvals. It receives adjusted proposal information as input and executes automation scripts and tasks as output. Specifically, automated task processing using RPA tools is scheduled on the server.

[0417] (Application Example 2)

[0418] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0419] Conventional business automation systems automate tasks without considering the user's emotional state, making it difficult to reduce the user's psychological burden. Repetitive administrative tasks, in particular, can increase stress, so task adjustments that take emotional states into account are needed.

[0420] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0421] In this invention, the server includes means for monitoring the user's work activities and collecting data; means for identifying repetitive administrative tasks from the collected data; means for generating automation suggestions for the identified administrative tasks and notifying the user; means for providing feedback on the emotional state via a display device as needed; means for analyzing the user's facial expressions and voice and evaluating their emotions; and means for making suggestions for improving work efficiency based on the emotional evaluation. This enables efficient work execution that takes into account the user's emotional state.

[0422] A "user" is the entity that utilizes the system and is the target of suggestions for automating repetitive administrative tasks and improving efficiency based on emotional evaluations through their work activities.

[0423] "Business activities" refer to the duties and tasks that users perform on a daily basis, and the actions that are subject to monitoring and data collection.

[0424] "Data" includes users' work activities, facial expressions, voice, and information derived therefrom, and is collected for the purpose of identifying repetitive clerical tasks and evaluating emotions.

[0425] "Office work" refers to tasks that are particularly routine and repetitive within business activities, and are therefore subject to automation.

[0426] "Automation suggestions" refer to system recommendations for improving the efficiency of identified administrative tasks.

[0427] A "display device" refers to hardware used to present information to a user, either visually or audibly.

[0428] "Emotional state" refers to the user's psychological state and is evaluated based on information such as facial expressions and voice.

[0429] "Emotional evaluation" refers to the process of analyzing a user's facial expressions and voice information to determine their psychological state.

[0430] "Efficiency suggestions" refer to specific recommendations made based on emotional evaluations to improve the user's work efficiency.

[0431] In embodiments of this invention, it is important to design a system including a server and a display device to support the user's work activities. First, the server monitors the user's work activities and collects data. This data includes operations on email, spreadsheets, project management applications, etc., that the user uses on a daily basis. The server uses this data to identify repetitive administrative tasks.

[0432] Furthermore, the server utilizes an emotion engine to evaluate the user's emotional state from their facial expressions and voice. Specifically, it employs speech recognition software and image analysis technology to analyze the user's voice tone and facial expressions in real time. For example, it uses Python and the OpenCV library, and the Hugging Face Transformers library for voice analysis. This allows the system to understand the user's emotional state and generate optimal suggestions for improving work efficiency based on that understanding.

[0433] If a user's stress level is high, the server can lower the priority of that task and suggest automating other, easier tasks. The generated suggestions are notified to the user via a display device, and the user can approve or adjust the suggestions.

[0434] For example, if a user's regular report-writing task is identified as a source of stress, the server might suggest "automating data collection for reports and only requiring annotations." Such suggestions help users perform their tasks efficiently with less emotional burden. An example of a prompt might be, "If a user is experiencing stress, what work adjustment suggestions would you offer?"

[0435] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0436] Step 1:

[0437] The server monitors user work activities and collects operational data such as emails and spreadsheets. This collected data is used later to identify repetitive administrative tasks. Inputs include real-time user operational data obtained from business software, and outputs are organized data streams.

[0438] Step 2:

[0439] The server analyzes the collected data and uses machine learning models to identify repetitive tasks. This identification process determines whether a particular task is repeated by recognizing patterns in the data. The input is the collected data stream, and the output is a list of identified tasks.

[0440] Step 3:

[0441] The server uses an emotion engine to analyze the user's facial expressions and voice to evaluate their emotional state. It utilizes speech recognition and image analysis technologies to assess how emotions are changing in real time. Specifically, it takes camera video and audio data as input and outputs emotion data as the analysis result.

[0442] Step 4:

[0443] The server generates automation suggestions for improving work efficiency based on identified administrative tasks and user sentiment data. A generative AI model, trained on historical data, is used to generate these suggestions. Inputs include a list of administrative tasks and sentiment data, while output is optimized automation suggestions.

[0444] Step 5:

[0445] The terminal displays automated suggestions sent from the server, allowing the user to approve and adjust the suggestions. The user can review these suggestions and customize them to their preferences. Suggestions are input from the display device, and the user's selections are recorded as the output of the operation.

[0446] Step 6:

[0447] If a user approves a proposal, the server automates the process based on the approved administrative task. The input includes the user-approved proposal, and the output is the automated workflow deployed within the system.

[0448] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0449] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0450] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0451] [Third Embodiment]

[0452] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0453] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0454] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0455] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0456] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0457] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0458] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0459] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0460] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0461] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0462] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0463] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0464] This invention provides an automation system for improving the work efficiency of users. This system aims to monitor users' work activities and automate repetitive tasks. Specific embodiments are described below.

[0465] Server Processing

[0466] The server establishes connections with the business tools that users use on a daily basis and monitors user activities. Examples include sending emails, editing spreadsheets, and managing tasks in project management tools. The server analyzes the data collected from these activities to identify repetitive tasks. For identified tasks, the server generates suggestions for automation. These suggestions include an overview of the efficiency gains from automation and the necessary configurations.

[0467] Terminal processing

[0468] The terminal displays automation proposals sent from the server to the user. The proposals include specific details of the automation, expected results, and any necessary customization information. When the user approves the proposal, the terminal sends that information back to the server, instructing it to start the automation process.

[0469] User processing

[0470] Users review automation suggestions they receive during their daily work. They consider the suggestions and approve them if they deem them useful. They can also customize the suggestions as needed. For example, they can adjust the target data range or change the execution timing to better fit their work.

[0471] As a concrete example, suppose a user is responsible for creating weekly team meeting minutes and sending them to everyone. The server identifies this task and generates a proposal to "automatically create meeting minutes from a template every Friday and email them to team members." Once the user approves this proposal, the server sets up the automated process and executes it automatically at the specified time. In this way, the user is freed from repetitive tasks and can focus on other important work.

[0472] The system of the present invention thus improves overall productivity by reducing the user's operational burden and optimizing business processes.

[0473] The following describes the processing flow.

[0474] Step 1:

[0475] The server establishes API connections with the user's business tools and monitors their daily work activities. This includes user email operations, spreadsheet editing, and task management in project management tools.

[0476] Step 2:

[0477] The server analyzes the collected business data and uses machine learning algorithms to identify repetitive tasks. In this process, pattern recognition techniques are used to identify operations that are repeated at a certain frequency.

[0478] Step 3:

[0479] Based on the identified tasks, the server evaluates the benefits of automation and generates automation proposals. These proposals detail areas where improvements in operational efficiency and time savings are expected.

[0480] Step 4:

[0481] The terminal displays automation suggestions sent from the server to the user on its interface. These suggestions include the tasks to be performed, the schedule, and customization options.

[0482] Step 5:

[0483] Users review the proposals via their terminal and provide input for approval or customization. Users can specify execution timing and conditions as needed.

[0484] Step 6:

[0485] The server builds automated workflows to fit specific business processes based on user-approved suggestions. At this stage, the necessary scripts and settings are automatically generated in the backend.

[0486] Step 7:

[0487] The server debugs the automatically generated code, detecting and correcting errors. This ensures the accuracy and stability of the code.

[0488] Step 8:

[0489] The server automatically executes the configured workflow according to a pre-configured schedule. The results of the execution are recorded and saved for evaluation.

[0490] Step 9:

[0491] The terminal notifies the user of the execution results. The success or failure status is displayed, allowing the user to re-execute or adjust the process as needed.

[0492] Step 10:

[0493] Users review the provided results and provide feedback as needed. This feedback is sent to the server and used to improve and optimize the system.

[0494] (Example 1)

[0495] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0496] In today's work environment, users often spend a significant amount of time on repetitive tasks, and there is a need to improve productivity. The challenge is to efficiently automate these repetitive tasks so that users can focus on more important work.

[0497] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0498] In this invention, the server includes means for monitoring the work activities of individual users using an information processing device and collecting operation logs, means for analyzing the collected operation logs to identify repetitive tasks, and means for creating automation suggestions for the identified tasks and communicating them to the user. This enables efficient execution of a process to automate repetitive tasks that users perform on a daily basis, thereby improving productivity.

[0499] An "information processing device" refers to a device that has the functions of inputting, processing, and outputting data, and is used to perform calculations and data manipulation.

[0500] "User" is a term that refers to an individual or group that performs specific operations using an information processing device, and is the ultimate beneficiary of the system.

[0501] "Work activity" refers to the specific operations or parts of a task that a user performs on an information processing device, and generally encompasses a series of actions required to complete a task.

[0502] An "operation log" refers to a collection of data recorded by an information processing device that shows the user's actions, and usually includes information that shows the user's behavior history.

[0503] "Analysis" refers to the process of using statistical or machine learning methods to elucidate the meaning and patterns of collected data.

[0504] "Repetitive tasks" refer to business processes in which the same operations are repeated over a certain period of time, and are usually targets for automation.

[0505] An "automation proposal" refers to a document that describes a series of procedures or methods devised to automate a specific operation, and is presented to the user.

[0506] A "processing procedure" refers to a set of sequential steps that a system follows to complete a specific task, and it forms the basis of automated operations.

[0507] "Program code" refers to a series of instructions written to cause an information processing device to perform a specific process.

[0508] "Means of fixing a problem" refers to methods for identifying the cause and implementing solutions when software or hardware does not function as intended.

[0509] "Execution result" refers to the output data generated as a result of instructions given by the user to the information processing device, and indicates whether the operation was successful or unsuccessful.

[0510] This invention is an automated system designed to improve the operational efficiency of users. The system is primarily implemented by three entities: the server, the terminal, and the user.

[0511] The server acts as an information processing device, establishing connections with business tools to effectively monitor user activity. Specifically, cloud-based email services, spreadsheet software, and task management applications are used. The server collects activity logs from these tools and analyzes them using machine learning algorithms. During the analysis, the server identifies repetitive tasks and generates suggestions for automation. This generation process utilizes cutting-edge artificial intelligence technology called generative AI models, which construct specific suggestions through prompt messages.

[0512] The terminal plays a crucial role in notifying users of automation proposals generated by the server. The proposal details, including the specific automation mechanism and expected deliverables, are displayed on the terminal's screen. Users can approve or modify the proposal via the terminal, and the results are then sent back to the server.

[0513] The user is the entity that actually performs the work and makes appropriate decisions based on the automation suggestions provided from the terminal. For example, if the user is responsible for creating meeting minutes after a regular weekly meeting and sending them to the relevant members, the server can suggest something like, "Automatically create and send meeting minutes every Friday using a template." This process is implemented using a prompt message like the following: "Please suggest a way for the user to automate the task of creating and sending meeting minutes after a regular weekly meeting."

[0514] By implementing this invention, users will be freed from repetitive tasks and will be able to concentrate on tasks that require higher-level judgment and creativity. The entire system is designed to reduce workload and improve efficiency through information processing.

[0515] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0516] Step 1:

[0517] The server establishes a connection with the user's business tools and monitors their work activities. Specifically, it accesses email clients and spreadsheet applications via APIs and obtains activity logs of the user's actions. Inputs include user activity logs, and output is log data stored in the server's database.

[0518] Step 2:

[0519] The server analyzes accumulated log data to identify recurring operations and tasks performed by users. Machine learning algorithms are used for the analysis, with log data provided as input. As output, the server identifies recurring work patterns and lists potential automation candidates based on them.

[0520] Step 3:

[0521] The server uses a generative AI model to generate automation proposals for the identified automation candidates. Prompt statements are used to generate proposals that describe specific automation steps and expected benefits. The input here is the identified automation candidates, and the output includes the generated proposals.

[0522] Step 4:

[0523] The terminal displays automation suggestions sent from the server to the user. The terminal screen presents the automation details and expected results for the user to review. The input is the automation suggestion sent from the server, and the output is a visual representation of the information conveyed to the user.

[0524] Step 5:

[0525] Users review the displayed proposals, modify the plan as needed, and finally approve it. The specific input is the proposal information displayed on the terminal, and the output is the approved or modified information sent to the server. Users operate this process using the terminal's interface.

[0526] Step 6:

[0527] The server receives approval or correction information from users and constructs the automated workflow. If necessary, it automatically generates program code and verifies that there are no operational problems. The input is user approval information, and the output is the configured automated process.

[0528] Step 7:

[0529] The server executes automated processes based on a schedule. For example, it can automatically send emails or generate documents at scheduled times. The input to the execution is the configured workflow, and the output is the completion status of the predetermined tasks reported to the user as a result of the execution.

[0530] (Application Example 1)

[0531] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0532] In modern operations, particularly in logistics centers and other work environments, there are many repetitive and time-consuming tasks. As a result, staff members spend a significant amount of time and effort on these tasks, hindering efficient operations. In this context, there is a need for effective systems that automate tasks and improve operational efficiency.

[0533] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0534] In this invention, the server includes means for monitoring the actions of the work performer and collecting information, means for identifying repetitive tasks from the collected information, and means for generating automation suggestions for the identified tasks and notifying the work performer. This enables the efficiency and automation of repetitive tasks in the workplace.

[0535] A "work performer" is the person who directly carries out the work and is the entity that makes decisions and performs operations related to the content of the work.

[0536] "Action" refers to a series of actions or work processes performed by a person carrying out a task.

[0537] "Information" refers to data and records obtained from the actions of those performing the work, and is used for analyzing the content of the work.

[0538] "Repetitive tasks" refer to work activities in which the same or similar actions are repeated regularly and continuously.

[0539] An "automation proposal" presents the person performing the work with specific methods and processes to streamline identified repetitive tasks.

[0540] "Notification" refers to the act of communicating automation suggestions or other information to those performing the work.

[0541] This invention provides a system for streamlining repetitive tasks in business sites such as logistics centers. This system monitors the actions of workers, identifies repetitive tasks from the information obtained, and generates and notifies users of suggestions for automation of those tasks.

[0542] The server monitors the actions of the work performers in real time and processes the collected information using data analysis software. This data analysis software incorporates machine learning algorithms, enabling high-precision identification of repetitive tasks. Based on the identified tasks, specific suggestions for automation are generated and communicated to the work performers. Visualization devices such as smart glasses and tablets are used in this process.

[0543] The person performing the task reviews the proposal displayed on the visualization device, customizes it as needed, and approves it. The approved proposal is sent to the server, which automatically generates a program and automates the work. This process allows the person performing the task to proceed more efficiently.

[0544] For example, if a logistics center staff member picks items from shelves along a similar route at a fixed time each day, this task can be identified, generating a proposal to "automate picking on this route at 10:00 AM every day." Once the staff member approves this proposal, a picking robot can perform the work along the automated route. This allows the staff member to focus on other important tasks.

[0545] An example of a prompt for a generating AI model is: "Generate prompts for the system to recognize the user's daily work events and identify parts that can be automated. For example, if the same task is repeated at a specific time, consider suggestions for automating it." This prompt provides the AI ​​model with the information necessary to generate automation suggestions.

[0546] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0547] Step 1:

[0548] The server monitors the actions of the worker through smart glasses or cameras and collects the video data. The input is real-time video data, and the output is the action information to be analyzed. In this step, the video data is temporarily stored in preparation for analysis in the next step.

[0549] Step 2:

[0550] The server processes the collected video data using data analysis software and identifies repetitive tasks using machine learning algorithms. The input is the behavioral information obtained in step 1, and the output is a list of identified repetitive tasks. Data analysis involves time-series data analysis and clustering to find specific patterns.

[0551] Step 3:

[0552] The server generates automation suggestions based on identified repetitive tasks. The input is the list of repetitive tasks obtained from step 2, and the output is the automation suggestions. A generative AI model is used for generation, and automation proposals are formulated based on past successes and criteria for operational efficiency.

[0553] Step 4:

[0554] The terminal notifies the task implementer of the generated automation proposals and displays them on a visualization device. The input is the automation proposals sent from the server, and the output is the notification to the task implementer. This operation allows the implementer to understand the content of the proposals.

[0555] Step 5:

[0556] The user reviews the automation proposal displayed on the terminal and performs customization or approval operations as needed. The input is the automation proposal from step 4, and the output is the approved proposal or customized content. Through this operation, the implementer makes adjustments according to specific work conditions.

[0557] Step 6:

[0558] The server receives approved proposals from users and automatically generates programs to prepare for automated task execution. The input is user approval information, and the output is automatically generated program code. A code generation tool is used for program generation, and debugging is performed concurrently during the generation process.

[0559] Step 7:

[0560] The server executes automatically generated programs to automate repetitive tasks. The input is the program code from step 6, and the output is the completed result of the automated work. Clear time management and resource allocation are implemented during this execution.

[0561] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0562] This invention combines an automated system for improving user work efficiency with an emotion engine that recognizes user emotions. The system aims not only to monitor user work activities and automate repetitive tasks, but also to generate suggestions that take the user's emotional state into account. Specific embodiments are described below.

[0563] Server Processing

[0564] The server monitors work activities through the business tools used by the user. This includes user interactions in email, spreadsheets, and project management applications. The server collects this activity data and uses machine learning algorithms to identify repetitive tasks.

[0565] Furthermore, the server utilizes an emotion engine to evaluate the user's emotions by analyzing non-verbal data such as the user's voice, facial expressions, and typing speed. This emotion data is then used to suggest automation improvements for increased work efficiency. For example, if a user is experiencing stress, the system will lower the priority of that task and generate suggestions to automate other tasks instead.

[0566] Terminal processing

[0567] The terminal displays automated suggestions sent from the server to the user. These suggestions include adjustments based on analysis by the emotion engine. For example, to reduce user stress, suggestions may be made to distribute the workload.

[0568] Users can review these proposals from their devices and approve or customize them. Approved proposals are sent to the server, and the automated workflow is built.

[0569] User processing

[0570] Users can receive automation suggestions and sentiment-based analysis results in real time, allowing them to adjust their work processes. For example, if a particular task is determined to be emotionally demanding, the user can reschedule that task.

[0571] As a concrete example, suppose a user repeatedly performs the task of creating daily reports. The server identifies this task as one that can be automated and also determines that the user is under stress based on their facial expression and slow input speed. In this case, the system reduces the user's burden by suggesting that "data collection for the daily reports be automated, and the user only edits the parts they need to edit."

[0572] Thus, the system of the present invention can optimize operations by taking into account not only the user's workload but also their psychological state, thereby providing a more effective work environment.

[0573] The following describes the processing flow.

[0574] Step 1:

[0575] The server integrates with users' business tools to monitor their daily work activities. Specifically, it acquires operation logs from email software, spreadsheets, project management tools, and other similar applications.

[0576] Step 2:

[0577] The server uses an emotion engine to collect nonverbal data in real time, such as the user's voice tone, facial expressions, and typing speed. This allows the server to evaluate the user's emotional state.

[0578] Step 3:

[0579] The server analyzes business data and uses machine learning algorithms to identify repetitive tasks. For example, this includes tasks such as writing reports that are performed at the same time every day.

[0580] Step 4:

[0581] Based on the analysis results from the emotion engine, the server adjusts task priorities and generates optimal automation suggestions for the user. Specifically, if the user is experiencing stress, suggestions will be made to reduce the workload of that task.

[0582] Step 5:

[0583] The terminal displays automated suggestions received from the server to the user via its interface. At this stage, the suggestions incorporate adjustments based on sentiment analysis.

[0584] Step 6:

[0585] Users can review the proposals via their device and approve or customize them. They can also set execution timings and specific conditions as needed.

[0586] Step 7:

[0587] The server builds a customized automated workflow based on the user's approved proposal, automatically creating the necessary scripts and task configurations.

[0588] Step 8:

[0589] The server executes the configured workflow according to the set schedule. After execution, it records the results and saves them to the database.

[0590] Step 9:

[0591] The terminal notifies the user of the execution results. This includes success, failure, and feedback collection functions as needed.

[0592] Step 10:

[0593] Users review the execution results and provide feedback. This information is sent to the server and used to improve the emotion engine and automation system.

[0594] (Example 2)

[0595] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0596] In today's business environment, many tasks performed by users involve repetitive work, which contributes to decreased work efficiency. Furthermore, stress and emotional burden can directly impact work performance. However, conventional automation systems have difficulty flexibly responding to users' emotional states, making it challenging to alleviate their psychological burden.

[0597] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0598] In this invention, the server includes means for monitoring the user's work activities via a processing device and collecting information; means for identifying repetitive tasks from the collected information and performing analysis using a machine learning algorithm; and means for analyzing the user's nonverbal information and adjusting the suggested content by evaluating their emotional state. This enables automated suggestions that take into account the user's emotional state, improving work efficiency and reducing emotional burden.

[0599] "Users" refers to individuals or groups who actually use the system and conduct business activities using it.

[0600] "Business activities" refers to the collective term for the operations and tasks performed by users in the course of their jobs.

[0601] A "processing device" refers to a mechanical or electronic device used to collect, analyze, store, or output data.

[0602] "Information" refers to content that the system acquires or generates for processing and analysis, such as users' work activities and non-verbal data.

[0603] "Repetitive tasks" refer to work tasks that involve the same procedures or processes occurring repeatedly.

[0604] A "machine learning algorithm" refers to a computational method that learns patterns from data and automatically performs analysis and prediction.

[0605] "Non-verbal information" refers to data other than linguistic information, such as the user's voice, facial expressions, and typing speed.

[0606] "Emotional state" refers to the user's psychological state, including stress levels, satisfaction levels, and motivation.

[0607] "Adjusting the proposed content" refers to modifying automated suggestions based on the user's emotional state and work situation.

[0608] This invention is implemented by a system consisting of a server, terminals, and users. The server first monitors business activities through business software used by the users. Specifically, the server uses a computing device equipped with a high-performance processor and large-capacity storage. The server collects data via APIs of software used by the users, such as email, spreadsheets, and project management software, and stores it in a database.

[0609] The server then runs machine learning algorithms using the collected data. This execution uses the Python language and the scikit-learn library, and includes cluster analysis to identify repetitive tasks.

[0610] Furthermore, the server uses an emotion engine to analyze non-verbal data such as the user's voice, facial expressions, and input speed. This utilizes speech recognition software and facial expression analysis libraries (e.g., OpenCV).

[0611] The data generated from the analysis will be used to generate automation suggestions in the form of prompts using a generative AI model (e.g., GPT-3). The aim of these suggestions is to reduce the user's workload and to make adjustments that take emotional states into consideration.

[0612] The terminal's role is to visually display automation suggestions received from the server to the user. This uses HTML and JavaScript to build the user interface.

[0613] Users can review these proposals via their devices and approve or adjust them. Approved proposals are sent back to the server, and the necessary workflows are automated.

[0614] To give a concrete example, in a user's daily report-writing task, the server suggests automating the data entry portion. This allows the user to focus on reviewing and editing the necessary parts of the report, saving time and effort.

[0615] An example of a prompt message is generated in the format, "To what extent will automating this task reduce user stress?"

[0616] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0617] Step 1:

[0618] The server collects user work activity data through the APIs of business software. Inputs include emails, spreadsheets, and project management tool operation data. This data is stored in a database, making it available for subsequent analysis processes. Specifically, the system is designed to periodically call the APIs to retrieve the latest work activity information.

[0619] Step 2:

[0620] The server executes machine learning algorithms using the collected business data. This step uses business activity data stored in a database as input. Clustering analysis is performed using the scikit-learn library to identify repetitive tasks. The output is a list of identified repetitive tasks. Real-time functionality is achieved by running a Python script as a scheduled job.

[0621] Step 3:

[0622] The server uses an emotion engine to analyze nonverbal data and evaluate the user's emotional state. Inputs include voice, facial expression data, and input speed. These data are analyzed using speech recognition software and OpenCV to obtain numerical data for each emotional state. The output is an evaluation result indicating the user's emotional state. Specifically, the server is configured to process video and audio data in real time.

[0623] Step 4:

[0624] The server generates automation suggestions using a generative AI model. This step uses identified repetitive tasks and emotional state evaluation results as input data. The generative AI model receives prompt text and outputs the optimal automation suggestion. Specifically, the generative AI model is provided on a cloud service, and suggestion generation is performed via an API.

[0625] Step 5:

[0626] The terminal displays automation suggestions received from the server to the user. It receives suggestion data from the server as input and displays the suggested content on the user interface as output. Specifically, it uses HTML and JavaScript to build a visually appealing interface, ensuring that the suggestions are intuitively understandable to the user.

[0627] Step 6:

[0628] Users review proposals via a terminal and approve or adjust them as needed. Input involves reviewing the proposal displayed on the terminal's interface and entering their own opinions and requests. Output is the adjusted proposal information, which is sent to the server. Specific actions involve fine-tuning the proposal using a mouse and keyboard.

[0629] Step 7:

[0630] The server prepares to execute automated workflows based on user approvals. It receives adjusted proposal information as input and executes automation scripts and tasks as output. Specifically, automated task processing using RPA tools is scheduled on the server.

[0631] (Application Example 2)

[0632] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0633] Conventional business automation systems automate tasks without considering the user's emotional state, making it difficult to reduce the user's psychological burden. Repetitive administrative tasks, in particular, can increase stress, so task adjustments that take emotional states into account are needed.

[0634] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0635] In this invention, the server includes means for monitoring the user's work activities and collecting data; means for identifying repetitive administrative tasks from the collected data; means for generating automation suggestions for the identified administrative tasks and notifying the user; means for providing feedback on the emotional state via a display device as needed; means for analyzing the user's facial expressions and voice and evaluating their emotions; and means for making suggestions for improving work efficiency based on the emotional evaluation. This enables efficient work execution that takes into account the user's emotional state.

[0636] A "user" is the entity that utilizes the system and is the target of suggestions for automating repetitive administrative tasks and improving efficiency based on emotional evaluations through their work activities.

[0637] "Business activities" refer to the duties and tasks that users perform on a daily basis, and the actions that are subject to monitoring and data collection.

[0638] "Data" includes users' work activities, facial expressions, voice, and information derived therefrom, and is collected for the purpose of identifying repetitive clerical tasks and evaluating emotions.

[0639] "Office work" refers to tasks that are particularly routine and repetitive within business activities, and are therefore subject to automation.

[0640] "Automation suggestions" refer to system recommendations for improving the efficiency of identified administrative tasks.

[0641] A "display device" refers to hardware used to present information to a user, either visually or audibly.

[0642] "Emotional state" refers to the user's psychological state and is evaluated based on information such as facial expressions and voice.

[0643] "Emotional evaluation" refers to the process of analyzing a user's facial expressions and voice information to determine their psychological state.

[0644] "Efficiency suggestions" refer to specific recommendations made based on emotional evaluations to improve the user's work efficiency.

[0645] In embodiments of this invention, it is important to design a system including a server and a display device to support the user's work activities. First, the server monitors the user's work activities and collects data. This data includes operations on email, spreadsheets, project management applications, etc., that the user uses on a daily basis. The server uses this data to identify repetitive administrative tasks.

[0646] Furthermore, the server utilizes an emotion engine to evaluate the user's emotional state from their facial expressions and voice. Specifically, it employs speech recognition software and image analysis technology to analyze the user's voice tone and facial expressions in real time. For example, it uses Python and the OpenCV library, and the Hugging Face Transformers library for voice analysis. This allows the system to understand the user's emotional state and generate optimal suggestions for improving work efficiency based on that understanding.

[0647] If a user's stress level is high, the server can lower the priority of that task and suggest automating other, easier tasks. The generated suggestions are notified to the user via a display device, and the user can approve or adjust the suggestions.

[0648] For example, if a user's regular report-writing task is identified as a source of stress, the server might suggest "automating data collection for reports and only requiring annotations." Such suggestions help users perform their tasks efficiently with less emotional burden. An example of a prompt might be, "If a user is experiencing stress, what work adjustment suggestions would you offer?"

[0649] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0650] Step 1:

[0651] The server monitors user work activities and collects operational data such as emails and spreadsheets. This collected data is used later to identify repetitive administrative tasks. Inputs include real-time user operational data obtained from business software, and outputs are organized data streams.

[0652] Step 2:

[0653] The server analyzes the collected data and uses machine learning models to identify repetitive tasks. This identification process determines whether a particular task is repeated by recognizing patterns in the data. The input is the collected data stream, and the output is a list of identified tasks.

[0654] Step 3:

[0655] The server uses an emotion engine to analyze the user's facial expressions and voice to evaluate their emotional state. It utilizes speech recognition and image analysis technologies to assess how emotions are changing in real time. Specifically, it takes camera video and audio data as input and outputs emotion data as the analysis result.

[0656] Step 4:

[0657] The server generates automation suggestions for improving work efficiency based on identified administrative tasks and user sentiment data. A generative AI model, trained on historical data, is used to generate these suggestions. Inputs include a list of administrative tasks and sentiment data, while output is optimized automation suggestions.

[0658] Step 5:

[0659] The terminal displays automated suggestions sent from the server, allowing the user to approve and adjust the suggestions. The user can review these suggestions and customize them to their preferences. Suggestions are input from the display device, and the user's selections are recorded as the output of the operation.

[0660] Step 6:

[0661] If a user approves a proposal, the server automates the process based on the approved administrative task. The input includes the user-approved proposal, and the output is the automated workflow deployed within the system.

[0662] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0663] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0664] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0665] [Fourth Embodiment]

[0666] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0667] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0668] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0669] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0670] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0671] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0672] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0673] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0674] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0675] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0676] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0677] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0678] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0679] This invention provides an automation system for improving the work efficiency of users. This system aims to monitor users' work activities and automate repetitive tasks. Specific embodiments are described below.

[0680] Server Processing

[0681] The server establishes connections with the business tools that users use on a daily basis and monitors user activities. Examples include sending emails, editing spreadsheets, and managing tasks in project management tools. The server analyzes the data collected from these activities to identify repetitive tasks. For identified tasks, the server generates suggestions for automation. These suggestions include an overview of the efficiency gains from automation and the necessary configurations.

[0682] Terminal processing

[0683] The terminal displays automation proposals sent from the server to the user. The proposals include specific details of the automation, expected results, and any necessary customization information. When the user approves the proposal, the terminal sends that information back to the server, instructing it to start the automation process.

[0684] User processing

[0685] Users review automation suggestions they receive during their daily work. They consider the suggestions and approve them if they deem them useful. They can also customize the suggestions as needed. For example, they can adjust the target data range or change the execution timing to better fit their work.

[0686] As a concrete example, suppose a user is responsible for creating weekly team meeting minutes and sending them to everyone. The server identifies this task and generates a proposal to "automatically create meeting minutes from a template every Friday and email them to team members." Once the user approves this proposal, the server sets up the automated process and executes it automatically at the specified time. In this way, the user is freed from repetitive tasks and can focus on other important work.

[0687] The system of the present invention thus improves overall productivity by reducing the user's operational burden and optimizing business processes.

[0688] The following describes the processing flow.

[0689] Step 1:

[0690] The server establishes API connections with the user's business tools and monitors their daily work activities. This includes user email operations, spreadsheet editing, and task management in project management tools.

[0691] Step 2:

[0692] The server analyzes the collected business data and uses machine learning algorithms to identify repetitive tasks. In this process, pattern recognition techniques are used to identify operations that are repeated at a certain frequency.

[0693] Step 3:

[0694] Based on the identified tasks, the server evaluates the benefits of automation and generates automation proposals. These proposals detail areas where improvements in operational efficiency and time savings are expected.

[0695] Step 4:

[0696] The terminal displays automation suggestions sent from the server to the user on its interface. These suggestions include the tasks to be performed, the schedule, and customization options.

[0697] Step 5:

[0698] Users review the proposals via their terminal and provide input for approval or customization. Users can specify execution timing and conditions as needed.

[0699] Step 6:

[0700] The server builds automated workflows to fit specific business processes based on user-approved suggestions. At this stage, the necessary scripts and settings are automatically generated in the backend.

[0701] Step 7:

[0702] The server debugs the automatically generated code, detecting and correcting errors. This ensures the accuracy and stability of the code.

[0703] Step 8:

[0704] The server automatically executes the configured workflow according to a pre-configured schedule. The results of the execution are recorded and saved for evaluation.

[0705] Step 9:

[0706] The terminal notifies the user of the execution results. The success or failure status is displayed, allowing the user to re-execute or adjust the process as needed.

[0707] Step 10:

[0708] Users review the provided results and provide feedback as needed. This feedback is sent to the server and used to improve and optimize the system.

[0709] (Example 1)

[0710] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0711] In today's work environment, users often spend a significant amount of time on repetitive tasks, and there is a need to improve productivity. The challenge is to efficiently automate these repetitive tasks so that users can focus on more important work.

[0712] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0713] In this invention, the server includes means for monitoring the work activities of individual users using an information processing device and collecting operation logs, means for analyzing the collected operation logs to identify repetitive tasks, and means for creating automation suggestions for the identified tasks and communicating them to the user. This enables efficient execution of a process to automate repetitive tasks that users perform on a daily basis, thereby improving productivity.

[0714] An "information processing device" refers to a device that has the functions of inputting, processing, and outputting data, and is used to perform calculations and data manipulation.

[0715] "User" is a term that refers to an individual or group that performs specific operations using an information processing device, and is the ultimate beneficiary of the system.

[0716] "Work activity" refers to the specific operations or parts of a task that a user performs on an information processing device, and generally encompasses a series of actions required to complete a task.

[0717] An "operation log" refers to a collection of data recorded by an information processing device that shows the user's actions, and usually includes information that shows the user's behavior history.

[0718] "Analysis" refers to the process of using statistical or machine learning methods to elucidate the meaning and patterns of collected data.

[0719] "Repetitive tasks" refer to business processes in which the same operations are repeated over a certain period of time, and are usually targets for automation.

[0720] An "automation proposal" refers to a document that describes a series of procedures or methods devised to automate a specific operation, and is presented to the user.

[0721] A "processing procedure" refers to a set of sequential steps that a system follows to complete a specific task, and it forms the basis of automated operations.

[0722] "Program code" refers to a series of instructions written to cause an information processing device to perform a specific process.

[0723] "Means of fixing a problem" refers to methods for identifying the cause and implementing solutions when software or hardware does not function as intended.

[0724] "Execution result" refers to the output data generated as a result of instructions given by the user to the information processing device, and indicates whether the operation was successful or unsuccessful.

[0725] This invention is an automated system designed to improve the operational efficiency of users. The system is primarily implemented by three entities: the server, the terminal, and the user.

[0726] The server acts as an information processing device, establishing connections with business tools to effectively monitor user activity. Specifically, cloud-based email services, spreadsheet software, and task management applications are used. The server collects activity logs from these tools and analyzes them using machine learning algorithms. During the analysis, the server identifies repetitive tasks and generates suggestions for automation. This generation process utilizes cutting-edge artificial intelligence technology called generative AI models, which construct specific suggestions through prompt messages.

[0727] The terminal plays a crucial role in notifying users of automation proposals generated by the server. The proposal details, including the specific automation mechanism and expected deliverables, are displayed on the terminal's screen. Users can approve or modify the proposal via the terminal, and the results are then sent back to the server.

[0728] The user is the entity that actually performs the work and makes appropriate decisions based on the automation suggestions provided from the terminal. For example, if the user is responsible for creating meeting minutes after a regular weekly meeting and sending them to the relevant members, the server can suggest something like, "Automatically create and send meeting minutes every Friday using a template." This process is implemented using a prompt message like the following: "Please suggest a way for the user to automate the task of creating and sending meeting minutes after a regular weekly meeting."

[0729] By implementing this invention, users will be freed from repetitive tasks and will be able to concentrate on tasks that require higher-level judgment and creativity. The entire system is designed to reduce workload and improve efficiency through information processing.

[0730] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0731] Step 1:

[0732] The server establishes a connection with the user's business tools and monitors their work activities. Specifically, it accesses email clients and spreadsheet applications via APIs and obtains activity logs of the user's actions. Inputs include user activity logs, and output is log data stored in the server's database.

[0733] Step 2:

[0734] The server analyzes accumulated log data to identify recurring operations and tasks performed by users. Machine learning algorithms are used for the analysis, with log data provided as input. As output, the server identifies recurring work patterns and lists potential automation candidates based on them.

[0735] Step 3:

[0736] The server uses a generative AI model to generate automation proposals for the identified automation candidates. Prompt statements are used to generate proposals that describe specific automation steps and expected benefits. The input here is the identified automation candidates, and the output includes the generated proposals.

[0737] Step 4:

[0738] The terminal displays automation suggestions sent from the server to the user. The terminal screen presents the automation details and expected results for the user to review. The input is the automation suggestion sent from the server, and the output is a visual representation of the information conveyed to the user.

[0739] Step 5:

[0740] Users review the displayed proposals, modify the plan as needed, and finally approve it. The specific input is the proposal information displayed on the terminal, and the output is the approved or modified information sent to the server. Users operate this process using the terminal's interface.

[0741] Step 6:

[0742] The server receives approval or correction information from users and constructs the automated workflow. If necessary, it automatically generates program code and verifies that there are no operational problems. The input is user approval information, and the output is the configured automated process.

[0743] Step 7:

[0744] The server executes automated processes based on a schedule. For example, it can automatically send emails or generate documents at scheduled times. The input to the execution is the configured workflow, and the output is the completion status of the predetermined tasks reported to the user as a result of the execution.

[0745] (Application Example 1)

[0746] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0747] In modern operations, particularly in logistics centers and other work environments, there are many repetitive and time-consuming tasks. As a result, staff members spend a significant amount of time and effort on these tasks, hindering efficient operations. In this context, there is a need for effective systems that automate tasks and improve operational efficiency.

[0748] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0749] In this invention, the server includes means for monitoring the actions of the work performer and collecting information, means for identifying repetitive tasks from the collected information, and means for generating automation suggestions for the identified tasks and notifying the work performer. This enables the efficiency and automation of repetitive tasks in the workplace.

[0750] A "work performer" is the person who directly carries out the work and is the entity that makes decisions and performs operations related to the content of the work.

[0751] "Action" refers to a series of actions or work processes performed by a person carrying out a task.

[0752] "Information" refers to data and records obtained from the actions of those performing the work, and is used for analyzing the content of the work.

[0753] "Repetitive tasks" refer to work activities in which the same or similar actions are repeated regularly and continuously.

[0754] An "automation proposal" presents the person performing the work with specific methods and processes to streamline identified repetitive tasks.

[0755] "Notification" refers to the act of communicating automation suggestions or other information to those performing the work.

[0756] This invention provides a system for streamlining repetitive tasks in business sites such as logistics centers. This system monitors the actions of workers, identifies repetitive tasks from the information obtained, and generates and notifies users of suggestions for automation of those tasks.

[0757] The server monitors the actions of the work performers in real time and processes the collected information using data analysis software. This data analysis software incorporates machine learning algorithms, enabling high-precision identification of repetitive tasks. Based on the identified tasks, specific suggestions for automation are generated and communicated to the work performers. Visualization devices such as smart glasses and tablets are used in this process.

[0758] The person performing the task reviews the proposal displayed on the visualization device, customizes it as needed, and approves it. The approved proposal is sent to the server, which automatically generates a program and automates the work. This process allows the person performing the task to proceed more efficiently.

[0759] For example, if a logistics center staff member picks items from shelves along a similar route at a fixed time each day, this task can be identified, generating a proposal to "automate picking on this route at 10:00 AM every day." Once the staff member approves this proposal, a picking robot can perform the work along the automated route. This allows the staff member to focus on other important tasks.

[0760] An example of a prompt for a generating AI model is: "Generate prompts for the system to recognize the user's daily work events and identify parts that can be automated. For example, if the same task is repeated at a specific time, consider suggestions for automating it." This prompt provides the AI ​​model with the information necessary to generate automation suggestions.

[0761] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0762] Step 1:

[0763] The server monitors the actions of the worker through smart glasses or cameras and collects the video data. The input is real-time video data, and the output is the action information to be analyzed. In this step, the video data is temporarily stored in preparation for analysis in the next step.

[0764] Step 2:

[0765] The server processes the collected video data using data analysis software and identifies repetitive tasks using machine learning algorithms. The input is the behavioral information obtained in step 1, and the output is a list of identified repetitive tasks. Data analysis involves time-series data analysis and clustering to find specific patterns.

[0766] Step 3:

[0767] The server generates automation suggestions based on identified repetitive tasks. The input is the list of repetitive tasks obtained from step 2, and the output is the automation suggestions. A generative AI model is used for generation, and automation proposals are formulated based on past successes and criteria for operational efficiency.

[0768] Step 4:

[0769] The terminal notifies the task implementer of the generated automation proposals and displays them on a visualization device. The input is the automation proposals sent from the server, and the output is the notification to the task implementer. This operation allows the implementer to understand the content of the proposals.

[0770] Step 5:

[0771] The user reviews the automation proposal displayed on the terminal and performs customization or approval operations as needed. The input is the automation proposal from step 4, and the output is the approved proposal or customized content. Through this operation, the implementer makes adjustments according to specific work conditions.

[0772] Step 6:

[0773] The server receives approved proposals from users and automatically generates programs to prepare for automated task execution. The input is user approval information, and the output is automatically generated program code. A code generation tool is used for program generation, and debugging is performed concurrently during the generation process.

[0774] Step 7:

[0775] The server executes automatically generated programs to automate repetitive tasks. The input is the program code from step 6, and the output is the completed result of the automated work. Clear time management and resource allocation are implemented during this execution.

[0776] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0777] This invention combines an automated system for improving user work efficiency with an emotion engine that recognizes user emotions. The system aims not only to monitor user work activities and automate repetitive tasks, but also to generate suggestions that take the user's emotional state into account. Specific embodiments are described below.

[0778] Server Processing

[0779] The server monitors work activities through the business tools used by the user. This includes user interactions in email, spreadsheets, and project management applications. The server collects this activity data and uses machine learning algorithms to identify repetitive tasks.

[0780] Furthermore, the server utilizes an emotion engine to evaluate the user's emotions by analyzing non-verbal data such as the user's voice, facial expressions, and typing speed. This emotion data is then used to suggest automation improvements for increased work efficiency. For example, if a user is experiencing stress, the system will lower the priority of that task and generate suggestions to automate other tasks instead.

[0781] Terminal processing

[0782] The terminal displays automated suggestions sent from the server to the user. These suggestions include adjustments based on analysis by the emotion engine. For example, to reduce user stress, suggestions may be made to distribute the workload.

[0783] Users can review these proposals from their devices and approve or customize them. Approved proposals are sent to the server, and the automated workflow is built.

[0784] User processing

[0785] Users can receive automation suggestions and sentiment-based analysis results in real time, allowing them to adjust their work processes. For example, if a particular task is determined to be emotionally demanding, the user can reschedule that task.

[0786] As a concrete example, suppose a user repeatedly performs the task of creating daily reports. The server identifies this task as one that can be automated and also determines that the user is under stress based on their facial expression and slow input speed. In this case, the system reduces the user's burden by suggesting that "data collection for the daily reports be automated, and the user only edits the parts they need to edit."

[0787] Thus, the system of the present invention can optimize operations by taking into account not only the user's workload but also their psychological state, thereby providing a more effective work environment.

[0788] The following describes the processing flow.

[0789] Step 1:

[0790] The server integrates with users' business tools to monitor their daily work activities. Specifically, it acquires operation logs from email software, spreadsheets, project management tools, and other similar applications.

[0791] Step 2:

[0792] The server uses an emotion engine to collect nonverbal data in real time, such as the user's voice tone, facial expressions, and typing speed. This allows the server to evaluate the user's emotional state.

[0793] Step 3:

[0794] The server analyzes business data and uses machine learning algorithms to identify repetitive tasks. For example, this includes tasks such as writing reports that are performed at the same time every day.

[0795] Step 4:

[0796] Based on the analysis results from the emotion engine, the server adjusts task priorities and generates optimal automation suggestions for the user. Specifically, if the user is experiencing stress, suggestions will be made to reduce the workload of that task.

[0797] Step 5:

[0798] The terminal displays automated suggestions received from the server to the user via its interface. At this stage, the suggestions incorporate adjustments based on sentiment analysis.

[0799] Step 6:

[0800] Users can review the proposals via their device and approve or customize them. They can also set execution timings and specific conditions as needed.

[0801] Step 7:

[0802] The server builds a customized automated workflow based on the user's approved proposal, automatically creating the necessary scripts and task configurations.

[0803] Step 8:

[0804] The server executes the configured workflow according to the set schedule. After execution, it records the results and saves them to the database.

[0805] Step 9:

[0806] The terminal notifies the user of the execution results. This includes success, failure, and feedback collection functions as needed.

[0807] Step 10:

[0808] Users review the execution results and provide feedback. This information is sent to the server and used to improve the emotion engine and automation system.

[0809] (Example 2)

[0810] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0811] In today's business environment, many tasks performed by users involve repetitive work, which contributes to decreased work efficiency. Furthermore, stress and emotional burden can directly impact work performance. However, conventional automation systems have difficulty flexibly responding to users' emotional states, making it challenging to alleviate their psychological burden.

[0812] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0813] In this invention, the server includes means for monitoring the user's work activities via a processing device and collecting information; means for identifying repetitive tasks from the collected information and performing analysis using a machine learning algorithm; and means for analyzing the user's nonverbal information and adjusting the suggested content by evaluating their emotional state. This enables automated suggestions that take into account the user's emotional state, improving work efficiency and reducing emotional burden.

[0814] "Users" refers to individuals or groups who actually use the system and conduct business activities using it.

[0815] "Business activities" refers to the collective term for the operations and tasks performed by users in the course of their jobs.

[0816] A "processing device" refers to a mechanical or electronic device used to collect, analyze, store, or output data.

[0817] "Information" refers to content that the system acquires or generates for processing and analysis, such as users' work activities and non-verbal data.

[0818] "Repetitive tasks" refer to work tasks that involve the same procedures or processes occurring repeatedly.

[0819] A "machine learning algorithm" refers to a computational method that learns patterns from data and automatically performs analysis and prediction.

[0820] "Non-verbal information" refers to data other than linguistic information, such as the user's voice, facial expressions, and typing speed.

[0821] "Emotional state" refers to the user's psychological state, including stress levels, satisfaction levels, and motivation.

[0822] "Adjusting the proposed content" refers to modifying automated suggestions based on the user's emotional state and work situation.

[0823] This invention is implemented by a system consisting of a server, terminals, and users. The server first monitors business activities through business software used by the users. Specifically, the server uses a computing device equipped with a high-performance processor and large-capacity storage. The server collects data via APIs of software used by the users, such as email, spreadsheets, and project management software, and stores it in a database.

[0824] The server then runs machine learning algorithms using the collected data. This execution uses the Python language and the scikit-learn library, and includes cluster analysis to identify repetitive tasks.

[0825] Furthermore, the server uses an emotion engine to analyze non-verbal data such as the user's voice, facial expressions, and input speed. This utilizes speech recognition software and facial expression analysis libraries (e.g., OpenCV).

[0826] The data generated from the analysis will be used to generate automation suggestions in the form of prompts using a generative AI model (e.g., GPT-3). The aim of these suggestions is to reduce the user's workload and to make adjustments that take emotional states into consideration.

[0827] The terminal's role is to visually display automation suggestions received from the server to the user. This uses HTML and JavaScript to build the user interface.

[0828] Users can review these proposals via their devices and approve or adjust them. Approved proposals are sent back to the server, and the necessary workflows are automated.

[0829] To give a concrete example, in a user's daily report-writing task, the server suggests automating the data entry portion. This allows the user to focus on reviewing and editing the necessary parts of the report, saving time and effort.

[0830] An example of a prompt message is generated in the format, "To what extent will automating this task reduce user stress?"

[0831] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0832] Step 1:

[0833] The server collects user work activity data through the APIs of business software. Inputs include emails, spreadsheets, and project management tool operation data. This data is stored in a database, making it available for subsequent analysis processes. Specifically, the system is designed to periodically call the APIs to retrieve the latest work activity information.

[0834] Step 2:

[0835] The server executes machine learning algorithms using the collected business data. This step uses business activity data stored in a database as input. Clustering analysis is performed using the scikit-learn library to identify repetitive tasks. The output is a list of identified repetitive tasks. Real-time functionality is achieved by running a Python script as a scheduled job.

[0836] Step 3:

[0837] The server uses an emotion engine to analyze nonverbal data and evaluate the user's emotional state. Inputs include voice, facial expression data, and input speed. These data are analyzed using speech recognition software and OpenCV to obtain numerical data for each emotional state. The output is an evaluation result indicating the user's emotional state. Specifically, the server is configured to process video and audio data in real time.

[0838] Step 4:

[0839] The server generates automation suggestions using a generative AI model. This step uses identified repetitive tasks and emotional state evaluation results as input data. The generative AI model receives prompt text and outputs the optimal automation suggestion. Specifically, the generative AI model is provided on a cloud service, and suggestion generation is performed via an API.

[0840] Step 5:

[0841] The terminal displays automation suggestions received from the server to the user. It receives suggestion data from the server as input and displays the suggested content on the user interface as output. Specifically, it uses HTML and JavaScript to build a visually appealing interface, ensuring that the suggestions are intuitively understandable to the user.

[0842] Step 6:

[0843] Users review proposals via a terminal and approve or adjust them as needed. Input involves reviewing the proposal displayed on the terminal's interface and entering their own opinions and requests. Output is the adjusted proposal information, which is sent to the server. Specific actions involve fine-tuning the proposal using a mouse and keyboard.

[0844] Step 7:

[0845] The server prepares to execute automated workflows based on user approvals. It receives adjusted proposal information as input and executes automation scripts and tasks as output. Specifically, automated task processing using RPA tools is scheduled on the server.

[0846] (Application Example 2)

[0847] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0848] Conventional business automation systems automate tasks without considering the user's emotional state, making it difficult to reduce the user's psychological burden. Repetitive administrative tasks, in particular, can increase stress, so task adjustments that take emotional states into account are needed.

[0849] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0850] In this invention, the server includes means for monitoring the user's work activities and collecting data; means for identifying repetitive administrative tasks from the collected data; means for generating automation suggestions for the identified administrative tasks and notifying the user; means for providing feedback on the emotional state via a display device as needed; means for analyzing the user's facial expressions and voice and evaluating their emotions; and means for making suggestions for improving work efficiency based on the emotional evaluation. This enables efficient work execution that takes into account the user's emotional state.

[0851] A "user" is the entity that utilizes the system and is the target of suggestions for automating repetitive administrative tasks and improving efficiency based on emotional evaluations through their work activities.

[0852] "Business activities" refer to the duties and tasks that users perform on a daily basis, and the actions that are subject to monitoring and data collection.

[0853] "Data" includes users' work activities, facial expressions, voice, and information derived therefrom, and is collected for the purpose of identifying repetitive clerical tasks and evaluating emotions.

[0854] "Office work" refers to tasks that are particularly routine and repetitive within business activities, and are therefore subject to automation.

[0855] "Automation suggestions" refer to system recommendations for improving the efficiency of identified administrative tasks.

[0856] A "display device" refers to hardware used to present information to a user, either visually or audibly.

[0857] "Emotional state" refers to the user's psychological state and is evaluated based on information such as facial expressions and voice.

[0858] "Emotional evaluation" refers to the process of analyzing a user's facial expressions and voice information to determine their psychological state.

[0859] "Efficiency suggestions" refer to specific recommendations made based on emotional evaluations to improve the user's work efficiency.

[0860] In embodiments of this invention, it is important to design a system including a server and a display device to support the user's work activities. First, the server monitors the user's work activities and collects data. This data includes operations on email, spreadsheets, project management applications, etc., that the user uses on a daily basis. The server uses this data to identify repetitive administrative tasks.

[0861] Furthermore, the server utilizes an emotion engine to evaluate the user's emotional state from their facial expressions and voice. Specifically, it employs speech recognition software and image analysis technology to analyze the user's voice tone and facial expressions in real time. For example, it uses Python and the OpenCV library, and the Hugging Face Transformers library for voice analysis. This allows the system to understand the user's emotional state and generate optimal suggestions for improving work efficiency based on that understanding.

[0862] If a user's stress level is high, the server can lower the priority of that task and suggest automating other, easier tasks. The generated suggestions are notified to the user via a display device, and the user can approve or adjust the suggestions.

[0863] For example, if a user's regular report-writing task is identified as a source of stress, the server might suggest "automating data collection for reports and only requiring annotations." Such suggestions help users perform their tasks efficiently with less emotional burden. An example of a prompt might be, "If a user is experiencing stress, what work adjustment suggestions would you offer?"

[0864] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0865] Step 1:

[0866] The server monitors user work activities and collects operational data such as emails and spreadsheets. This collected data is used later to identify repetitive administrative tasks. Inputs include real-time user operational data obtained from business software, and outputs are organized data streams.

[0867] Step 2:

[0868] The server analyzes the collected data and uses machine learning models to identify repetitive tasks. This identification process determines whether a particular task is repeated by recognizing patterns in the data. The input is the collected data stream, and the output is a list of identified tasks.

[0869] Step 3:

[0870] The server uses an emotion engine to analyze the user's facial expressions and voice to evaluate their emotional state. It utilizes speech recognition and image analysis technologies to assess how emotions are changing in real time. Specifically, it takes camera video and audio data as input and outputs emotion data as the analysis result.

[0871] Step 4:

[0872] The server generates automation suggestions for improving work efficiency based on identified administrative tasks and user sentiment data. A generative AI model, trained on historical data, is used to generate these suggestions. Inputs include a list of administrative tasks and sentiment data, while output is optimized automation suggestions.

[0873] Step 5:

[0874] The terminal displays automated suggestions sent from the server, allowing the user to approve and adjust the suggestions. The user can review these suggestions and customize them to their preferences. Suggestions are input from the display device, and the user's selections are recorded as the output of the operation.

[0875] Step 6:

[0876] If a user approves a proposal, the server automates the process based on the approved administrative task. The input includes the user-approved proposal, and the output is the automated workflow deployed within the system.

[0877] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0878] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0879] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0880] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0881] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0882] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0883] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0884] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0885] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0886] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0887] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0888] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0889] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0890] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0891] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0892] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0893] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0894] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0895] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0896] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0897] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0898] The following is further disclosed regarding the embodiments described above.

[0899] (Claim 1)

[0900] Means for monitoring users' work activities and collecting data,

[0901] A means of identifying repetitive tasks from collected data,

[0902] A means of generating automation suggestions for identified tasks and notifying the user,

[0903] A means of building a workflow to automate approved tasks after a user has approved a proposal,

[0904] A means of automatically generating and debugging code as needed,

[0905] A means of executing automated tasks and reporting the results to the user,

[0906] A system that includes this.

[0907] (Claim 2)

[0908] The system according to claim 1, further comprising means for enabling customization of automated suggestions based on user input.

[0909] (Claim 3)

[0910] The system according to claim 1, further comprising means for collecting user feedback and using it to improve the system.

[0911] "Example 1"

[0912] (Claim 1)

[0913] A means of monitoring the work activities of individual users and collecting operation logs using an information processing device,

[0914] A means of analyzing collected operation logs to identify recurring tasks,

[0915] A means of creating and communicating automation proposals for identified tasks to users,

[0916] A means to set up a processing procedure to automate the approved task when the user approves the proposal,

[0917] A means to automatically generate program code as needed and correct problems,

[0918] A means of performing automated tasks and reporting the results to the user,

[0919] A system that includes this.

[0920] (Claim 2)

[0921] The system according to claim 1, which enables modification of automated suggestions based on user input.

[0922] (Claim 3)

[0923] The system according to claim 1, further comprising means for collecting user feedback and utilizing it to improve the system.

[0924] "Application Example 1"

[0925] (Claim 1)

[0926] Means for monitoring the actions of those performing the work and collecting information,

[0927] A means of identifying repetitive tasks from collected information,

[0928] A means of generating automation proposals for identified tasks and notifying the task implementers,

[0929] A means of building a processing procedure to automate the approved work after the person performing the work has approved the proposal,

[0930] A means to automatically generate and verify programs as needed,

[0931] A means of performing automated tasks and reporting the results to the person performing the task,

[0932] A means of displaying automation suggestions through a visualization device,

[0933] A means of performing automated tasks with the permission of the person performing the work,

[0934] A system that includes this.

[0935] (Claim 2)

[0936] The system according to claim 1, further comprising means for enabling the identification of automation proposals based on input from the person performing the work.

[0937] (Claim 3)

[0938] The system according to claim 1, further comprising means for collecting feedback from those who perform the work and using it to improve the system.

[0939] "Example 2 of combining an emotion engine"

[0940] (Claim 1)

[0941] A means of monitoring and collecting information on the user's work activities via a processing device,

[0942] A means of identifying repetitive tasks from collected information and performing analysis using machine learning algorithms,

[0943] A means of creating automation proposals for identified tasks and presenting them to the user,

[0944] A means of adjusting the proposed content by analyzing the user's nonverbal information and evaluating their emotional state,

[0945] A means for generating processing steps to automate approved work after the user has approved or adjusted the proposal,

[0946] A means to automatically generate and verify executable code as needed,

[0947] A means of performing automated tasks and reporting the results to the user,

[0948] A system that includes this.

[0949] (Claim 2)

[0950] The system according to claim 1, which enables the adjustment of automated suggestions using an emotion engine based on user information.

[0951] (Claim 3)

[0952] The system according to claim 1, which collects user feedback and uses it to optimize the system.

[0953] "Application example 2 when combining with an emotional engine"

[0954] (Claim 1)

[0955] Means for monitoring users' work activities and collecting data,

[0956] A means of identifying repetitive clerical tasks from collected data,

[0957] A means of generating automation suggestions for identified administrative tasks and notifying the user,

[0958] A means of building a process to automate approved administrative tasks after a user has approved a proposal,

[0959] A means of providing feedback on emotional state via a display device as needed,

[0960] A method for analyzing users' facial expressions and voices to evaluate their emotions,

[0961] A means of proposing work efficiency improvements based on emotional evaluation,

[0962] A means of performing automated tasks and reporting the results to the user,

[0963] A system that includes this.

[0964] (Claim 2)

[0965] The system according to claim 1, further comprising means for enabling customization of automated suggestions based on user input.

[0966] (Claim 3)

[0967] The system according to claim 1, further comprising means for collecting user feedback and using it to improve the system. [Explanation of Symbols]

[0968] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means for monitoring the actions of those performing the work and collecting information, A means of identifying repetitive tasks from collected information, A means of generating automation proposals for identified tasks and notifying the task implementers, A means of building a processing procedure to automate the approved work after the person performing the work has approved the proposal, A means to automatically generate and verify programs as needed, A means of performing automated tasks and reporting the results to the person performing the task, A means of displaying automation suggestions through a visualization device, A means of performing automated tasks with the permission of the person performing the work, A system that includes this.

2. The system according to claim 1, further comprising means for enabling the identification of automation proposals based on input from the person performing the work.

3. The system according to claim 1, further comprising means for collecting feedback from those who perform the work and using it to improve the system.

Images