system

Abstract

We provide the system. [Solution] Means for obtaining user usage trends, A means for identifying repetitive administrative tasks based on the aforementioned usage trends, A means for generating instructions to automate the aforementioned repetitive administrative tasks, A means of providing the aforementioned instructions to the user and performing automation, A means of monitoring the progress of an organization's work and detecting anomalies, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a modern business environment, many workers spend a lot of time and effort on repetitive routine tasks, resulting in a problem that the time spent on creative and strategic tasks is restricted. Also, in the business progress and resource allocation of the whole organization, it is difficult to achieve real-time visualization and early detection of abnormalities, which may lead to progress delays and inefficient resource utilization. To solve these problems, a system that supports the automation and efficiency improvement of individual and whole-organization business is necessary.

Means for Solving the Problems

[0005] This invention provides a system that includes means for acquiring user usage trends, means for identifying repetitive administrative tasks based on usage trends, means for generating instructions for automating repetitive administrative tasks, means for providing instructions to users and automating the tasks, and means for monitoring the organization's work progress and detecting anomalies, thereby achieving automation and efficiency improvements in business operations. Furthermore, by including means for analyzing past work data and generating work improvement suggestions, and means for analyzing movement patterns in real time and proposing efficient movement routes, it enables the provision of an even more efficient work environment.

[0006] "Usage trends" refer to the patterns of operations and applications that users frequently perform in their daily work.

[0007] "Office work" refers to routine tasks, including operations and processes performed according to established procedures within the business.

[0008] "Automation" refers to the process of automatically handling tasks that were previously performed manually by humans, using machines or systems.

[0009] "Instructions" refer to commands that a system uses to perform specific operations or processes.

[0010] "An anomaly" refers to a state in which unexpected events or problems occur during normal business operations or resource usage.

[0011] "Progress" refers to an indicator that shows how far along a task or project is in relation to the plan.

[0012] "Efficiency" refers to optimizing the time and resources required for tasks and work to achieve maximum results.

[0013] "Travel patterns" refer to the tendencies in the routes and methods that users use when traveling for work.

[0014] "Visualization" refers to representing information and data in visual forms such as graphs and charts to make them easier to understand.

Brief Description of the Drawings

[0015] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [[ID=3৮]] [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Modes for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention provides a system for achieving automation and efficiency improvements in business operations. The system consists of a terminal and a server working in conjunction.

[0037] The terminal has a client program installed to monitor and record user actions. This program collects real-time information on user actions, application usage, and location during daily work, and sends the data to a server.

[0038] The server receives data sent from the terminal and runs a program to analyze user usage patterns. Specifically, it uses machine learning algorithms to identify repetitive office tasks and movement patterns. Based on this analysis, the server generates instructions to automate repetitive tasks.

[0039] Meanwhile, the server also runs a program that monitors the organization's work progress. This program visualizes the progress of tasks and has the function of detecting anomalies. When an anomaly is detected, the server sends an alert to the user and suggests necessary improvements.

[0040] The user receives instructions from the server and executes the automation. For example, the server can generate a script that automatically opens files the user should open daily and notify the user, allowing the user to work more efficiently. The server can also analyze past data and make new suggestions for improving work processes.

[0041] Through this series of operations, the present invention achieves the automation of business processes and improves operational efficiency for users.

[0042] The following describes the processing flow.

[0043] Step 1:

[0044] The device records the user's activity history in real time. This includes keyboard and mouse input, application launches and shutdowns, usage time, and document opening and closing. It also obtains movement history using the smartphone's location services.

[0045] Step 2:

[0046] The device encrypts the data collected at regular intervals and sends it to the server. Data transmission is performed using a secure protocol.

[0047] Step 3:

[0048] The server stores the received data and stores it in a database. The data is organized by user and prepared for future analysis.

[0049] Step 4:

[0050] The server analyzes the accumulated data using machine learning algorithms. This analysis identifies repetitive operations frequently performed by users and routines carried out for specific purposes.

[0051] Step 5:

[0052] Based on the analysis results, the server generates scripts to automate identified routines. For example, if a user opens the same file at the same time every day, it will create a script to automate that operation.

[0053] Step 6:

[0054] The server notifies the user of the generated scripts and suggested business improvements. At this stage, the user can review the proposed automations and approve or reject them.

[0055] Step 7:

[0056] If the user approves the script, the automation is executed by running the script on the device. This frees the user from monotonous tasks.

[0057] Step 8:

[0058] The server monitors the progress of tasks across the entire organization in real time. Through the task management dashboard, it visualizes the progress and resource usage of each member and automatically issues alerts if any anomalies are detected.

[0059] Step 9:

[0060] Users and teams make adjustments to improve operational efficiency based on feedback and suggestions from the server. This, in turn, increases the productivity of the entire organization.

[0061] (Example 1)

[0062] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0063] As the need for business efficiency and automation increases, it is essential to appropriately identify and automate repetitive tasks in the daily work activities of users. Furthermore, optimizing operations by understanding the overall progress of operations within the organization and quickly detecting anomalies is a challenge. Additionally, it is necessary to generate suggestions for continuous business improvement by utilizing past business data.

[0064] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0065] In this invention, the server includes means for monitoring user operations and collecting information, means for using computational models to analyze the data, and means for generating instructions to automate repetitive tasks. This enables increased efficiency and automation of operations. Furthermore, by monitoring the progress of the organization's operations and providing warnings and work improvement suggestions based on anomaly detection, it becomes possible to optimize operations and achieve continuous improvement.

[0066] "Monitoring user operations" means recording inputs and actions on a terminal in order to understand the work activities and intentions of the user.

[0067] "Collecting information" means gathering and organizing data for a specific purpose and using it for subsequent analysis and processing.

[0068] A "computational model" is a program based on mathematical or statistical methods used for data analysis, and is a model used for pattern extraction and prediction.

[0069] "Business activities" generally refer to a series of actions and processes performed in the course of carrying out business.

[0070] "Automating" means minimizing human intervention and having software or machines perform specific tasks.

[0071] "Monitoring work progress" means observing the progress of work to keep track of its status and confirming that it is proceeding according to plan.

[0072] "Anomaly detection" refers to identifying data or situations that deviate from the normal range and pointing out areas that require attention.

[0073] "Warning" refers to providing notifications or alerts to draw the user's attention.

[0074] A "work improvement proposal" is a suggestion for reforms or specific methods to improve efficiency in business operations.

[0075] This invention is a system that achieves automation and efficiency in business operations, and is built through the collaboration of terminals and servers. Specific embodiments are shown below.

[0076] The terminal has a client program installed to monitor user actions and collect information. This program records user work-related actions in real time and collects data on location and applications used. The collected data is sent to the server in an appropriate format.

[0077] The server is responsible for analyzing the received data. The processes running on the server include computational models for data analysis based on user actions (for example, machine learning models built in a Python environment). These models extract patterns of business activities from the collected data and identify recurring business activities. Based on these results, automation instructions are generated. These instructions are then provided to the user from the server, enabling the execution of automated business processes.

[0078] As a concrete example, the server generates a script to automatically open files that a user should open at a specific time each day. The user receives this instruction and can streamline their work through the automated process. Furthermore, the server can leverage historical data to suggest new business improvements, contributing to continuous operational efficiency.

[0079] Generative AI models are highly effective in recognizing patterns in such business activities and generating automation instructions. An example of a prompt is: "Propose an algorithm that analyzes the applications and operation patterns used by the user daily, and based on that, generates automation instructions aimed at improving work efficiency." This prompt serves as a guideline for accurately analyzing the user's intent and maximizing the potential of automation.

[0080] This system aims to speed up operations, improve accuracy, and enhance convenience and efficiency for users.

[0081] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0082] Step 1:

[0083] The terminal monitors user operations in real time and collects data on work-related activities. Specifically, it collects mouse clicks, keyboard input, application usage logs, and location information. This data is recorded as a time-series operation log. The collected data is processed into a predefined format by the client program and prepared for transmission to the server.

[0084] Step 2:

[0085] The terminal periodically sends collected operation log data to the server. This data is encrypted to ensure secure transfer. The transmitted data includes the user ID, timestamp, and details of each action. This data is stored on the server as raw material for analysis.

[0086] Step 3:

[0087] The server receives the incoming data and analyzes it using a machine learning computational model. Specifically, it uses the Python pandas library to format the data and clean up outliers. Next, it extracts patterns of specific business activities through analysis using scikit-learn. This analysis allows for the identification of recurring business activities.

[0088] Step 4:

[0089] The server generates automation instructions based on patterns derived from the analysis. For example, it can generate scripts to automatically open files that need to be opened at a specific time each day, or templates for sending standard emails. These instructions are generated as Python scripts or JavaScript® and are ready to be provided to the user.

[0090] Step 5:

[0091] The user receives and executes automation instructions sent from the server. In this process, the user runs a provided script, automating a portion of their work. This automated action allows the user to reduce the effort required for routine daily tasks.

[0092] Step 6:

[0093] The server monitors the overall operational progress of the organization. Data is visualized based on task progress, and delays are detected by an anomaly detection algorithm. In response to detected anomalies, the server sends alerts to users via email or notifications to support more efficient work execution.

[0094] Step 7:

[0095] The server analyzes past business data and proposes business improvement measures. For example, it analyzes the steps users took to perform tasks and generates suggestions for more efficient methods or the introduction of new tools. These suggestions are notified to users and incorporated into business processes.

[0096] (Application Example 1)

[0097] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0098] There is a need for a system that improves the operational efficiency of work equipment and robots used in factories, enabling automation and progress monitoring of tasks, and allowing for immediate response in the event of an anomaly. This will facilitate collaborative work with workers and improve productivity.

[0099] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0100] In this invention, the server includes means for monitoring and recording user operations, means for visualizing the progress of tasks based on those operations, and means for generating optimization instructions and displaying them on a visual device. This makes it possible to automatically improve the operational efficiency of work equipment within a factory and immediately improve operations.

[0101] "Means for monitoring and recording user operations" refers to technology that allows a system to check in real time the operation and behavior of equipment by a specific user and save that information as a history.

[0102] "Methods for visualizing work progress" refer to technologies that display the progress of work in an easily understandable format, enabling users and managers to quickly grasp the degree of progress and any delays in the work.

[0103] "Means for generating optimization instructions and displaying them on a visual device" refers to a technology that analyzes efficient operating methods in equipment and work environments, and provides a function to encourage improved operation by visually presenting the results to the user.

[0104] To implement this invention, a system is constructed that links a server and a terminal. The terminal has a program installed to monitor and record user operations, collecting daily work and equipment operation information in real time and sending the data to the server. This program continuously tracks the actions performed by the user and stores them as a history.

[0105] The server uses Python and Scikit-learn to analyze received data and runs a program to visualize the progress of tasks. This allows for immediate warnings if any anomalies are detected or if the work is progressing as planned. The server also generates optimal operational instructions and displays them on a terminal or visual device (e.g., ANDROID® smart glasses) using Unity. Through this visual device, the server presents the user with visualized instructions for specific work procedures.

[0106] As a concrete example, in parts maintenance work within a factory, the inspection process of parts, which is normally performed manually, can be significantly streamlined by using efficient process instructions automatically generated from the robot's motion patterns. Users can improve the accuracy and speed of their work by relying on visual information displayed on smart glasses.

[0107] This allows us to prompt the generated AI model and utilize instruction statements such as, "Develop an application that monitors the operation of factory robots in real time, suggests optimal actions to improve efficiency, and visually presents specific instructions."

[0108] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0109] Step 1:

[0110] The terminal monitors user actions and collects action data in real time. Input is user action information, and output is a sequential history of those actions. The terminal program records the action details and prepares them for transmission to the server.

[0111] Step 2:

[0112] The server receives operation data sent from the terminal and stores it in a database. The input is the operation data from the terminal, and the output is the record stored in the database. By storing it in the database, the server efficiently manages the operation history.

[0113] Step 3:

[0114] The server analyzes stored data to detect operational trends and anomalies. The input is the operation history read from the database, and the output is trend analysis and anomaly warning information as a result of the analysis. It uses Scikit-learn to model the data and identify outliers and patterns.

[0115] Step 4:

[0116] The server generates optimized operation instructions and sends them to the visual device. The input is the analysis result, and the output is the visual operation instructions. Based on the analysis result, steps to optimize the process are determined, and instructions are generated to be displayed on the smart glasses via Unity.

[0117] Step 5:

[0118] The user performs tasks by following instructions from a visual device. The input is the display instructions from the visual device, and the output is the result of the performed task. The user sees the steps presented on the smart glasses and performs the tasks specifically, efficiently advancing their work.

[0119] Step 6:

[0120] The server receives the work results and feeds that information back into the next analysis. The input is the work results performed by the user, and the output is improvement data for the next analysis. Based on this feedback, the server provides prompts to the generating AI model to further improve efficiency.

[0121] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0122] The present invention is implemented in a form that combines emotion recognition functionality with a system that supports the automation and efficiency of business operations. The system consists of a terminal, a server, and an emotion engine.

[0123] The terminal has a client program installed that monitors and records the user's operation history and application usage. The terminal is also equipped with a camera and microphone, through which an emotion engine recognizes the user's emotional state in real time. The camera analyzes facial expressions using facial recognition technology, and the microphone performs voice analysis.

[0124] User operation data and emotional data are sent from the terminal to the server. The server receives this data and analyzes the user's usage patterns and emotional state. The emotional data recognized by the emotion engine is analyzed using machine learning algorithms to detect changes in the user's stress level and motivation.

[0125] Based on the analysis results, the server generates scripts to automate repetitive tasks and suggestions to improve work efficiency. In addition, it has a function to suggest task priorities and notify users of the need for breaks, taking into account the impact of the user's emotional state on work efficiency.

[0126] For example, if the emotion engine detects that a user is experiencing increased stress while performing a particular task, the server will generate an automated script for that task and suggest automating it in the future. Furthermore, it will improve the work environment by sending a notification to the user recommending that they pause the task and take a short break.

[0127] This system allows users to optimize their work environment in a way that suits their individual needs, thereby reducing physical and mental strain while improving productivity.

[0128] The following describes the processing flow.

[0129] Step 1:

[0130] The device monitors user activity and records input data, application usage, and information on opened and closed files. It also uses its built-in camera and microphone to analyze user facial expressions and voice tone, collecting emotional data.

[0131] Step 2:

[0132] The device encrypts the collected operational and emotional data and periodically sends it to the server. This ensures that data is processed while protecting user privacy.

[0133] Step 3:

[0134] The server stores the received data and analyzes it using machine learning algorithms. The purpose of the analysis is to identify patterns in user usage, recurring work processes, and emotional states.

[0135] Step 4:

[0136] The server generates scripts for business processes that can be automated based on the analysis results. At the same time, it considers the user's emotional state and generates suggestions for appropriate breaks and tasks if signs of stress or fatigue are detected.

[0137] Step 5:

[0138] The server notifies the user of the generated automation scripts and suggestions. The user can review them and choose whether to accept the automation. They can also make a similar decision regarding the suggested breaks.

[0139] Step 6:

[0140] If a user approves the automation, a script is executed on the terminal, automating the specified tasks. This frees users from tedious administrative work and improves work efficiency.

[0141] Step 7:

[0142] The server continuously monitors the overall progress of operations and evaluates the workload of the entire organization. If an anomaly is detected, it automatically sends an alert and notifies the administrator.

[0143] Step 8:

[0144] Users and administrators can use alerts and suggestions from the server to improve business processes and optimize resources. This is expected to lead to sustained productivity improvements.

[0145] (Example 2)

[0146] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0147] In today's work environment, users are required to efficiently handle diverse tasks and improve work efficiency while reducing physical and mental burden. However, conventional automation systems often struggle to dynamically optimize work efficiency while considering the user's emotional state, or to suggest optimal break times, thus failing to maximize overall work productivity.

[0148] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0149] In this invention, the server includes means for monitoring the user's operation history and acquiring usage trends, means for identifying recurring tasks based on usage trends and emotional states, and means for analyzing the user's emotional state in real time using emotion recognition technology. This enables the automation of dynamic tasks, adjustment of efficiency priorities, and suggestion of optimal break times, all while considering the user's emotional state.

[0150] "User operation history" refers to a record of operations performed by the user on the device, including data such as which applications were used and to what extent.

[0151] "Usage patterns" refer to patterns in how and how often users use a particular application or feature.

[0152] "Emotional state" refers to psychological elements such as stress levels and motivation exhibited by users during work, and is acquired through data analysis using cameras and microphones.

[0153] "Repetitive tasks" refer to routine tasks and routine work that users perform frequently, and are therefore suitable for automation.

[0154] "Adjusting work priorities" refers to the process of rearranging tasks that need to be done based on the user's emotional state and work situation, thereby supporting efficient work execution.

[0155] "Break timing suggestions" refers to a function that notifies users of the optimal timing for taking a break, taking into account their physical and mental burden.

[0156] "Emotion recognition technology" refers to technology that uses cameras and microphones to analyze facial expressions and voices in real time and infer the emotional state of the user.

[0157] This invention is a system that supports the automation and efficiency of business operations, and consists of a terminal, a server, and an emotion engine. Specifically, the terminal holds a client program for acquiring the user's operation history and operates the emotion engine through a camera and microphone. The camera analyzes the user's facial expressions using facial recognition technology, and the microphone performs voice analysis. Through these devices, the user's emotional state can be recognized in real time.

[0158] Operational and emotional data transmitted from the device are aggregated on a server. The server analyzes this data to identify the user's usage patterns and emotional state. In particular, emotional data obtained by the emotion engine is processed using machine learning algorithms to help detect changes in the user's stress levels and motivation.

[0159] Based on the analysis results, the server generates automation scripts and suggestions to help users streamline their work. For example, if the emotion engine determines that a user is stressed by a particular task, it will suggest a script to automate that task. It can also notify the user of appropriate break times, thereby improving the work environment.

[0160] As a concrete example, if a user detects high stress while creating a document, the server will send a notification such as, "We suggest automating the task using a document template. A 5-minute break is recommended." A concrete example of a prompt message for the generating AI model would be, "If the stress level rises, generate an automation script for the task and suggest a break."

[0161] This approach provides concrete implementation methods for optimizing the user's work environment, reducing physical and mental burden, and increasing productivity.

[0162] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0163] Step 1:

[0164] The device monitors the user's operation history and records usage. Specifically, it receives input such as the name of the application the user is using, the order of operations, and the duration of those operations, and stores this data in a database. This provides basic data for understanding user usage trends.

[0165] Step 2:

[0166] Using a camera and microphone attached to the device, the emotion engine recognizes the user's emotional state in real time. The camera acquires feature points of the user's face as input and analyzes facial expressions using facial recognition technology. The microphone collects voice input and analyzes voice tone and speaking speed. Based on these analysis results, emotional state data is output.

[0167] Step 3:

[0168] The terminal sends the acquired operation history data and emotional state data to the server. Here, the data is formatted and encrypted before being output as transmitted data.

[0169] Step 4:

[0170] The server analyzes the received operation history data and emotional state data. Based on the input data, it utilizes machine learning models to identify user usage patterns and changes in emotional state. It performs data calculations and outputs analysis results showing each user's stress level and changes in motivation.

[0171] Step 5:

[0172] Based on the analysis results, the server generates suggestions to improve the user's work efficiency. It creates automation scripts for repetitive tasks and suggests breaks or more efficient work processes for stressful tasks. These suggestions are output as prompts and notified to the user.

[0173] Step 6:

[0174] The user receives a notification and decides whether to accept the suggested automation script. This input is returned to the terminal, which updates the work procedure. This process ultimately improves the user's work environment.

[0175] (Application Example 2)

[0176] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0177] Many tasks and household chores that users face daily at home and in the workplace are frequently repeated, and there is a need for increased efficiency. Furthermore, because work methods that take into account the emotional state of users are not readily available, these tasks can lead to significant mental and physical burden and decreased work efficiency. Therefore, it is desirable to appropriately support and streamline work and household tasks in accordance with the emotional state of users, thereby reducing their mental and physical burden.

[0178] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0179] In this invention, the server includes means for acquiring user usage patterns, means for generating instructions to automate repetitive administrative tasks, and means for recognizing the user's emotional state in real time and proposing support activities based on that state. This enables efficient and effective work support tailored to the individual user's condition.

[0180] "Means for acquiring user usage trends" refers to a device or method that records the operations and selections made by a user and collects those patterns as data.

[0181] "Means for identifying repetitive administrative tasks" refers to a device or method that analyzes the aforementioned usage trend data and identifies routine tasks performed on a daily basis.

[0182] "Means for generating instructions to automate clerical tasks" refers to a device or method for constructing specific processes or procedures for efficiently automating identified repetitive tasks.

[0183] "Means for recognizing a user's emotional state in real time" refers to a device or method that uses a camera and microphone to analyze a user's facial expressions and voice to understand their emotions and grasp their psychological state at any given time.

[0184] "Means of proposing support activities" refers to a device or method that presents the most appropriate actions or services based on the user's emotional state and provides support that meets the user's needs.

[0185] In implementing this invention, the system mainly consists of a terminal, a server, an emotion engine, and a home robot.

[0186] The device has a client program installed to acquire the user's usage patterns. The device is equipped with a camera and a microphone, and the emotion engine recognizes the user's emotional state in real time through these devices. The camera is used to analyze facial expressions using facial recognition technology, and the microphone is used for voice analysis.

[0187] User operation data and emotional data are sent from the terminal to the server. The server receives this data and analyzes the user's usage patterns and emotional state. In particular, emotional data recognized by the emotion engine is analyzed using machine learning algorithms to detect changes in the user's stress level and motivation. Based on the user's emotional state and usage patterns, the server suggests support activities for work and household chores.

[0188] Home robots support the daily lives of families through physical assistance and conversation, based on instructions for support activities sent from a server. For example, if someone in the family is experiencing stress, the robot can analyze their condition and play relaxation music or suggest taking a break.

[0189] For example, if the server detects increased stress levels in a user while they are performing a task, it can suggest automated task processing to reduce stress in the future. Also, if a child feels stressed while doing homework, the robot might suggest, "Why don't you take a short break? Shall I get you a drink?"

[0190] An example of a prompt is, "Please tell me how to analyze the emotional state of family members and suggest support activities within the home as needed."

[0191] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0192] Step 1:

[0193] The device uses its camera and microphone to capture the user's facial expressions and voice in real time. The input consists of camera video and audio data. Facial expressions are analyzed from the camera video using OpenCV's facial recognition technology, and the audio data is analyzed using Google's Speech-to-Text API. The output consists of analyzed facial expression data and audio-to-text data.

[0194] Step 2:

[0195] The device analyzes facial expression data and voice-to-text data and sends it to the server. The data is sent to the cloud server via a secure communication protocol. The input is facial expression data and voice-to-text data, and this data is sent directly to the server as output.

[0196] Step 3:

[0197] The server analyzes the facial expression data and voice / text data it receives and uses a machine learning algorithm to evaluate the user's emotional state. This analysis determines the degree of stress and motivation the user is experiencing. The input is facial expression data and voice / text data, and the output is evaluation data indicating the emotional state.

[0198] Step 4:

[0199] The server generates appropriate support activities based on the user's emotional state. It determines the actions necessary for the user's stress relief and efficient task completion. The input is emotional state evaluation data, and the output is instruction data indicating the content of the support activity.

[0200] Step 5:

[0201] The server generates instruction data for support activities and sends it to the home robot, which then carries it out. This is done through seamless communication over the network, with the robot interacting with the user through voice and actions. The input is the instruction data sent from the server, and the output is the specific support actions performed by the robot within the home.

[0202] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0203] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0204] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0205] [Second Embodiment]

[0206] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0207] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0208] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0209] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0210] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0211] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0212] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0213] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0214] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0215] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0216] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0217] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0218] This invention provides a system for achieving automation and efficiency improvements in business operations. The system consists of a terminal and a server working in conjunction.

[0219] The terminal has a client program installed to monitor and record user actions. This program collects real-time information on user actions, application usage, and location during daily work, and sends the data to a server.

[0220] The server receives data sent from the terminal and runs a program to analyze user usage patterns. Specifically, it uses machine learning algorithms to identify repetitive office tasks and movement patterns. Based on this analysis, the server generates instructions to automate repetitive tasks.

[0221] Meanwhile, the server also runs a program that monitors the organization's work progress. This program visualizes the progress of tasks and has the function of detecting anomalies. When an anomaly is detected, the server sends an alert to the user and suggests necessary improvements.

[0222] The user receives instructions from the server and executes the automation. For example, the server can generate a script that automatically opens files the user should open daily and notify the user, allowing the user to work more efficiently. The server can also analyze past data and make new suggestions for improving work processes.

[0223] Through this series of operations, the present invention achieves the automation of business processes and improves operational efficiency for users.

[0224] The following describes the processing flow.

[0225] Step 1:

[0226] The device records the user's activity history in real time. This includes keyboard and mouse input, application launches and shutdowns, usage time, and document opening and closing. It also obtains movement history using the smartphone's location services.

[0227] Step 2:

[0228] The device encrypts the data collected at regular intervals and sends it to the server. Data transmission is performed using a secure protocol.

[0229] Step 3:

[0230] The server stores the received data and stores it in a database. The data is organized by user and prepared for future analysis.

[0231] Step 4:

[0232] The server analyzes the accumulated data using machine learning algorithms. This analysis identifies repetitive operations frequently performed by users and routines carried out for specific purposes.

[0233] Step 5:

[0234] Based on the analysis results, the server generates scripts to automate identified routines. For example, if a user opens the same file at the same time every day, it will create a script to automate that operation.

[0235] Step 6:

[0236] The server notifies the user of the generated scripts and suggested business improvements. At this stage, the user can review the proposed automations and approve or reject them.

[0237] Step 7:

[0238] If the user approves the script, the automation is executed by running the script on the device. This frees the user from monotonous tasks.

[0239] Step 8:

[0240] The server monitors the progress of tasks across the entire organization in real time. Through the task management dashboard, it visualizes the progress and resource usage of each member and automatically issues alerts if any anomalies are detected.

[0241] Step 9:

[0242] Users and teams make adjustments to improve operational efficiency based on feedback and suggestions from the server. This, in turn, increases the productivity of the entire organization.

[0243] (Example 1)

[0244] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0245] As the need for business efficiency and automation increases, it is essential to appropriately identify and automate repetitive tasks in the daily work activities of users. Furthermore, optimizing operations by understanding the overall progress of operations within the organization and quickly detecting anomalies is a challenge. Additionally, it is necessary to generate suggestions for continuous business improvement by utilizing past business data.

[0246] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0247] In this invention, the server includes means for monitoring user operations and collecting information, means for using computational models to analyze the data, and means for generating instructions to automate repetitive tasks. This enables increased efficiency and automation of operations. Furthermore, by monitoring the progress of the organization's operations and providing warnings and work improvement suggestions based on anomaly detection, it becomes possible to optimize operations and achieve continuous improvement.

[0248] "Monitoring user operations" means recording inputs and actions on a terminal in order to understand the work activities and intentions of the user.

[0249] "Collecting information" means gathering and organizing data for a specific purpose and using it for subsequent analysis and processing.

[0250] A "computational model" is a program based on mathematical or statistical methods used for data analysis, and is a model used for pattern extraction and prediction.

[0251] "Business activities" generally refer to a series of actions and processes performed in the course of carrying out business.

[0252] "Automating" means minimizing human intervention and having software or machines perform specific tasks.

[0253] "Monitoring work progress" means observing the progress of work to keep track of its status and confirming that it is proceeding according to plan.

[0254] "Anomaly detection" refers to identifying data or situations that deviate from the normal range and pointing out areas that require attention.

[0255] "Warning" refers to providing notifications or alerts to draw the user's attention.

[0256] A "work improvement proposal" is a suggestion for reforms or specific methods to improve efficiency in business operations.

[0257] This invention is a system that achieves automation and efficiency in business operations, and is built through the collaboration of terminals and servers. Specific embodiments are shown below.

[0258] The terminal has a client program installed to monitor user actions and collect information. This program records user work-related actions in real time and collects data on location and applications used. The collected data is sent to the server in an appropriate format.

[0259] The server is responsible for analyzing the received data. The processes running on the server include computational models for data analysis based on user actions (for example, machine learning models built in a Python environment). These models extract patterns of business activities from the collected data and identify recurring business activities. Based on these results, automation instructions are generated. These instructions are then provided to the user from the server, enabling the execution of automated business processes.

[0260] As a concrete example, the server generates a script to automatically open files that a user should open at a specific time each day. The user receives this instruction and can streamline their work through the automated process. Furthermore, the server can leverage historical data to suggest new business improvements, contributing to continuous operational efficiency.

[0261] Generative AI models are highly effective in recognizing patterns in such business activities and generating automation instructions. An example of a prompt is: "Propose an algorithm that analyzes the applications and operation patterns used by the user daily, and based on that, generates automation instructions aimed at improving work efficiency." This prompt serves as a guideline for accurately analyzing the user's intent and maximizing the potential of automation.

[0262] This system aims to speed up operations, improve accuracy, and enhance convenience and efficiency for users.

[0263] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0264] Step 1:

[0265] The terminal monitors user operations in real time and collects data on work-related activities. Specifically, it collects mouse clicks, keyboard input, application usage logs, and location information. This data is recorded as a time-series operation log. The collected data is processed into a predefined format by the client program and prepared for transmission to the server.

[0266] Step 2:

[0267] The terminal periodically sends collected operation log data to the server. This data is encrypted to ensure secure transfer. The transmitted data includes the user ID, timestamp, and details of each action. This data is stored on the server as raw material for analysis.

[0268] Step 3:

[0269] The server receives the incoming data and analyzes it using a machine learning computational model. Specifically, it uses the Python pandas library to format the data and clean up outliers. Next, it extracts patterns of specific business activities through analysis using scikit-learn. This analysis allows for the identification of recurring business activities.

[0270] Step 4:

[0271] The server generates automation instructions based on patterns derived from the analysis. For example, it might generate a script to automatically open files that need to be opened at a specific time each day, or a template for sending standard emails. These instructions are generated as Python or JavaScript scripts and ready to be provided to the user.

[0272] Step 5:

[0273] The user receives and executes automation instructions sent from the server. In this process, the user runs a provided script, automating a portion of their work. This automated action allows the user to reduce the effort required for routine daily tasks.

[0274] Step 6:

[0275] The server monitors the overall operational progress of the organization. Data is visualized based on task progress, and delays are detected by an anomaly detection algorithm. In response to detected anomalies, the server sends alerts to users via email or notifications to support more efficient work execution.

[0276] Step 7:

[0277] The server analyzes past business data and proposes business improvement measures. For example, it analyzes the steps users took to perform tasks and generates suggestions for more efficient methods or the introduction of new tools. These suggestions are notified to users and incorporated into business processes.

[0278] (Application Example 1)

[0279] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0280] There is a need for a system that improves the operational efficiency of work equipment and robots used in factories, enabling automation and progress monitoring of tasks, and allowing for immediate response in the event of an anomaly. This will facilitate collaborative work with workers and improve productivity.

[0281] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0282] In this invention, the server includes means for monitoring and recording the operations of users, means for visualizing the progress of operations-based tasks, and means for generating optimization instructions and displaying them on a visual device. Thereby, it is possible to automatically improve the movement efficiency of working devices in the factory and immediately improve operations.

[0283] The "means for monitoring and recording the operations of users" is a technology for the system to confirm in real time the operations and movements of devices by specific users and save that information as a history.

[0284] The "means for visualizing the progress of tasks" is a technology for displaying the progress of work in a form that is easy to visually recognize, enabling users and administrators to quickly grasp the progress and any delays in the work.

[0285] The "means for generating optimization instructions and displaying them on a visual device" is a technology that provides a function to promote the improvement of operations by analyzing efficient operation methods in devices and the working environment and visually presenting the results to the user.

[0286] To implement this invention, a system that coordinates the server and the terminal is constructed. A program for monitoring and recording the operations of users is installed on the terminal, which collects daily operations and device operation information in real time and transmits the data to the server. In this program, the actions performed by the user are continuously tracked and saved as a history.

[0287] The server uses Python and Scikit-learn to analyze the received data and execute a program for visualizing the progress of tasks. Thereby, warnings can be provided immediately when the progress or abnormalities of the work are detected. In addition, the server generates optimal operation instructions and displays them on the terminal or a visual device (e.g., Android smart glasses) using Unity. Through this visual device, specific work procedures are presented to the user as visualized instructions.

[0288] As a concrete example, in parts maintenance work within a factory, the inspection process of parts, which is normally performed manually, can be significantly streamlined by using efficient process instructions automatically generated from the robot's motion patterns. Users can improve the accuracy and speed of their work by relying on visual information displayed on smart glasses.

[0289] This allows us to prompt the generated AI model and utilize instruction statements such as, "Develop an application that monitors the operation of factory robots in real time, suggests optimal actions to improve efficiency, and visually presents specific instructions."

[0290] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0291] Step 1:

[0292] The terminal monitors user actions and collects action data in real time. Input is user action information, and output is a sequential history of those actions. The terminal program records the action details and prepares them for transmission to the server.

[0293] Step 2:

[0294] The server receives operation data sent from the terminal and stores it in a database. The input is the operation data from the terminal, and the output is the record stored in the database. By storing it in the database, the server efficiently manages the operation history.

[0295] Step 3:

[0296] The server analyzes stored data to detect operational trends and anomalies. The input is the operation history read from the database, and the output is trend analysis and anomaly warning information as a result of the analysis. It uses Scikit-learn to model the data and identify outliers and patterns.

[0297] Step 4:

[0298] The server generates optimized operation instructions and sends them to the visual device. The input is the analysis result, and the output is the visual operation instructions. Based on the analysis result, steps to optimize the process are determined, and instructions are generated to be displayed on the smart glasses via Unity.

[0299] Step 5:

[0300] The user performs tasks by following instructions from a visual device. The input is the display instructions from the visual device, and the output is the result of the performed task. The user sees the steps presented on the smart glasses and performs the tasks specifically, efficiently advancing their work.

[0301] Step 6:

[0302] The server receives the work results and feeds that information back into the next analysis. The input is the work results performed by the user, and the output is improvement data for the next analysis. Based on this feedback, the server provides prompts to the generating AI model to further improve efficiency.

[0303] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0304] The present invention is implemented in a form that combines emotion recognition functionality with a system that supports the automation and efficiency of business operations. The system consists of a terminal, a server, and an emotion engine.

[0305] The terminal has a client program installed that monitors and records the user's operation history and application usage. The terminal is also equipped with a camera and microphone, through which an emotion engine recognizes the user's emotional state in real time. The camera analyzes facial expressions using facial recognition technology, and the microphone performs voice analysis.

[0306] The user's operation data and emotional data are sent from the terminal to the server. The server receives these data and analyzes the user's usage trends and emotional states. The emotional data recognized by the emotion engine is analyzed using machine learning algorithms to detect changes in the user's stress level and motivation.

[0307] Based on the analysis results, the server generates an automated script for repetitive tasks and proposals to improve work efficiency. In addition, it has a function to notify proposals for work priorities and the need for breaks, considering the impact of the user's emotional state on work efficiency.

[0308] As a specific example, when the emotion engine detects that the user is under increasing stress while performing a specific task, the server generates an automatic processing script for that task and proposes to automate it in subsequent times. Furthermore, the business environment is improved by sending a notification to the user recommending interruption of the task and taking a short break.

[0309] With this system, the user can achieve efficiency suitable for their individual work environment, reduce physical and mental burdens, and improve productivity.

[0310] The following describes the processing flow.

[0311] Step 1:

[0312] The terminal monitors the user's operations and records input data, application usage status, and information on opened and closed files. Also, using the camera and microphone installed on the terminal, it analyzes the user's expressions and voice tones to collect emotional data.

[0313] Step 2:

[0314] The terminal encrypts the collected operation data and emotional data and sends it to the server periodically. This enables data processing while protecting the user's privacy.

[0315] Step 3:

[0316] The server stores the received data and analyzes it using machine learning algorithms. The purpose of the analysis is to identify patterns in user usage, recurring work processes, and emotional states.

[0317] Step 4:

[0318] The server generates scripts for business processes that can be automated based on the analysis results. At the same time, it considers the user's emotional state and generates suggestions for appropriate breaks and tasks if signs of stress or fatigue are detected.

[0319] Step 5:

[0320] The server notifies the user of the generated automation scripts and suggestions. The user can review them and choose whether to accept the automation. They can also make a similar decision regarding the suggested breaks.

[0321] Step 6:

[0322] If a user approves the automation, a script is executed on the terminal, automating the specified tasks. This frees users from tedious administrative work and improves work efficiency.

[0323] Step 7:

[0324] The server continuously monitors the overall progress of operations and evaluates the workload of the entire organization. If an anomaly is detected, it automatically sends an alert and notifies the administrator.

[0325] Step 8:

[0326] Users and administrators can use alerts and suggestions from the server to improve business processes and optimize resources. This is expected to lead to sustained productivity improvements.

[0327] (Example 2)

[0328] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0329] In today's work environment, users are required to efficiently handle diverse tasks and improve work efficiency while reducing physical and mental burden. However, conventional automation systems often struggle to dynamically optimize work efficiency while considering the user's emotional state, or to suggest optimal break times, thus failing to maximize overall work productivity.

[0330] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0331] In this invention, the server includes means for monitoring the user's operation history and acquiring usage trends, means for identifying recurring tasks based on usage trends and emotional states, and means for analyzing the user's emotional state in real time using emotion recognition technology. This enables the automation of dynamic tasks, adjustment of efficiency priorities, and suggestion of optimal break times, all while considering the user's emotional state.

[0332] "User operation history" refers to a record of operations performed by the user on the device, including data such as which applications were used and to what extent.

[0333] "Usage patterns" refer to patterns in how and how often users use a particular application or feature.

[0334] "Emotional state" refers to psychological elements such as stress levels and motivation exhibited by users during work, and is acquired through data analysis using cameras and microphones.

[0335] "Repetitive tasks" refer to routine tasks and routine work that users perform frequently, and are therefore suitable for automation.

[0336] "Adjusting work priorities" refers to the process of rearranging tasks that need to be done based on the user's emotional state and work situation, thereby supporting efficient work execution.

[0337] "Break timing suggestions" refers to a function that notifies users of the optimal timing for taking a break, taking into account their physical and mental burden.

[0338] "Emotion recognition technology" refers to technology that uses cameras and microphones to analyze facial expressions and voices in real time and infer the emotional state of the user.

[0339] This invention is a system that supports the automation and efficiency of business operations, and consists of a terminal, a server, and an emotion engine. Specifically, the terminal holds a client program for acquiring the user's operation history and operates the emotion engine through a camera and microphone. The camera analyzes the user's facial expressions using facial recognition technology, and the microphone performs voice analysis. Through these devices, the user's emotional state can be recognized in real time.

[0340] Operational and emotional data transmitted from the device are aggregated on a server. The server analyzes this data to identify the user's usage patterns and emotional state. In particular, emotional data obtained by the emotion engine is processed using machine learning algorithms to help detect changes in the user's stress levels and motivation.

[0341] Based on the analysis results, the server generates automation scripts and suggestions to help users streamline their work. For example, if the emotion engine determines that a user is stressed by a particular task, it will suggest a script to automate that task. It can also notify the user of appropriate break times, thereby improving the work environment.

[0342] As a concrete example, if a user detects high stress while creating a document, the server will send a notification such as, "We suggest automating the task using a document template. A 5-minute break is recommended." A concrete example of a prompt message for the generating AI model would be, "If the stress level rises, generate an automation script for the task and suggest a break."

[0343] This approach provides concrete implementation methods for optimizing the user's work environment, reducing physical and mental burden, and increasing productivity.

[0344] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0345] Step 1:

[0346] The device monitors the user's operation history and records usage. Specifically, it receives input such as the name of the application the user is using, the order of operations, and the duration of those operations, and stores this data in a database. This provides basic data for understanding user usage trends.

[0347] Step 2:

[0348] Using a camera and microphone attached to the device, the emotion engine recognizes the user's emotional state in real time. The camera acquires feature points of the user's face as input and analyzes facial expressions using facial recognition technology. The microphone collects voice input and analyzes voice tone and speaking speed. Based on these analysis results, emotional state data is output.

[0349] Step 3:

[0350] The terminal sends the acquired operation history data and emotional state data to the server. Here, the data is formatted and encrypted before being output as transmitted data.

[0351] Step 4:

[0352] The server analyzes the received operation history data and emotional state data. Based on the input data, it utilizes machine learning models to identify user usage patterns and changes in emotional state. It performs data calculations and outputs analysis results showing each user's stress level and changes in motivation.

[0353] Step 5:

[0354] Based on the analysis results, the server generates suggestions to improve the user's work efficiency. It creates automation scripts for repetitive tasks and suggests breaks or more efficient work processes for stressful tasks. These suggestions are output as prompts and notified to the user.

[0355] Step 6:

[0356] The user receives a notification and decides whether to accept the suggested automation script. This input is returned to the terminal, which updates the work procedure. This process ultimately improves the user's work environment.

[0357] (Application Example 2)

[0358] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0359] Many tasks and household chores that users face daily at home and in the workplace are frequently repeated, and there is a need for increased efficiency. Furthermore, because work methods that take into account the emotional state of users are not readily available, these tasks can lead to significant mental and physical burden and decreased work efficiency. Therefore, it is desirable to appropriately support and streamline work and household tasks in accordance with the emotional state of users, thereby reducing their mental and physical burden.

[0360] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0361] In this invention, the server includes means for acquiring user usage patterns, means for generating instructions to automate repetitive administrative tasks, and means for recognizing the user's emotional state in real time and proposing support activities based on that state. This enables efficient and effective work support tailored to the individual user's condition.

[0362] "Means for acquiring user usage trends" refers to a device or method that records the operations and selections made by a user and collects those patterns as data.

[0363] "Means for identifying repetitive administrative tasks" refers to a device or method that analyzes the aforementioned usage trend data and identifies routine tasks performed on a daily basis.

[0364] "Means for generating instructions to automate clerical tasks" refers to a device or method for constructing specific processes or procedures for efficiently automating identified repetitive tasks.

[0365] "Means for recognizing a user's emotional state in real time" refers to a device or method that uses a camera and microphone to analyze a user's facial expressions and voice to understand their emotions and grasp their psychological state at any given time.

[0366] "Means of proposing support activities" refers to a device or method that presents the most appropriate actions or services based on the user's emotional state and provides support that meets the user's needs.

[0367] In implementing this invention, the system mainly consists of a terminal, a server, an emotion engine, and a home robot.

[0368] The device has a client program installed to acquire the user's usage patterns. The device is equipped with a camera and a microphone, and the emotion engine recognizes the user's emotional state in real time through these devices. The camera is used to analyze facial expressions using facial recognition technology, and the microphone is used for voice analysis.

[0369] User operation data and emotional data are sent from the terminal to the server. The server receives this data and analyzes the user's usage patterns and emotional state. In particular, emotional data recognized by the emotion engine is analyzed using machine learning algorithms to detect changes in the user's stress level and motivation. Based on the user's emotional state and usage patterns, the server suggests support activities for work and household chores.

[0370] Home robots support the daily lives of families through physical assistance and conversation, based on instructions for support activities sent from a server. For example, if someone in the family is experiencing stress, the robot can analyze their condition and play relaxation music or suggest taking a break.

[0371] For example, if the server detects increased stress levels in a user while they are performing a task, it can suggest automated task processing to reduce stress in the future. Also, if a child feels stressed while doing homework, the robot might suggest, "Why don't you take a short break? Shall I get you a drink?"

[0372] An example of a prompt is, "Please tell me how to analyze the emotional state of family members and suggest support activities within the home as needed."

[0373] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0374] Step 1:

[0375] The device uses its camera and microphone to capture the user's facial expressions and voice in real time. The input consists of camera video and audio data. Facial expressions are analyzed from the camera video using OpenCV's facial recognition technology, and the audio data is analyzed using the Google Speech-to-Text API. The output consists of analyzed facial expression data and audio-to-text data.

[0376] Step 2:

[0377] The device analyzes facial expression data and voice-to-text data and sends it to the server. The data is sent to the cloud server via a secure communication protocol. The input is facial expression data and voice-to-text data, and this data is sent directly to the server as output.

[0378] Step 3:

[0379] The server analyzes the facial expression data and voice / text data it receives and uses a machine learning algorithm to evaluate the user's emotional state. This analysis determines the degree of stress and motivation the user is experiencing. The input is facial expression data and voice / text data, and the output is evaluation data indicating the emotional state.

[0380] Step 4:

[0381] The server generates appropriate support activities based on the user's emotional state. It determines the actions necessary for the user's stress relief and efficient task completion. The input is emotional state evaluation data, and the output is instruction data indicating the content of the support activity.

[0382] Step 5:

[0383] The server generates instruction data for support activities and sends it to the home robot, which then carries it out. This is done through seamless communication over the network, with the robot interacting with the user through voice and actions. The input is the instruction data sent from the server, and the output is the specific support actions performed by the robot within the home.

[0384] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0385] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0386] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0387] [Third Embodiment]

[0388] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0389] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0390] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0391] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0392] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0393] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0394] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0395] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0396] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0397] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0398] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0399] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0400] This invention provides a system for achieving automation and efficiency improvements in business operations. The system consists of a terminal and a server working in conjunction.

[0401] The terminal has a client program installed to monitor and record user actions. This program collects real-time information on user actions, application usage, and location during daily work, and sends the data to a server.

[0402] The server receives data sent from the terminal and runs a program to analyze user usage patterns. Specifically, it uses machine learning algorithms to identify repetitive office tasks and movement patterns. Based on this analysis, the server generates instructions to automate repetitive tasks.

[0403] Meanwhile, the server also runs a program that monitors the organization's work progress. This program visualizes the progress of tasks and has the function of detecting anomalies. When an anomaly is detected, the server sends an alert to the user and suggests necessary improvements.

[0404] The user receives instructions from the server and executes the automation. For example, the server can generate a script that automatically opens files the user should open daily and notify the user, allowing the user to work more efficiently. The server can also analyze past data and make new suggestions for improving work processes.

[0405] Through this series of operations, the present invention achieves the automation of business processes and improves operational efficiency for users.

[0406] The following describes the processing flow.

[0407] Step 1:

[0408] The device records the user's activity history in real time. This includes keyboard and mouse input, application launches and shutdowns, usage time, and document opening and closing. It also obtains movement history using the smartphone's location services.

[0409] Step 2:

[0410] The device encrypts the data collected at regular intervals and sends it to the server. Data transmission is performed using a secure protocol.

[0411] Step 3:

[0412] The server stores the received data and stores it in a database. The data is organized by user and prepared for future analysis.

[0413] Step 4:

[0414] The server analyzes the accumulated data using machine learning algorithms. This analysis identifies repetitive operations frequently performed by users and routines carried out for specific purposes.

[0415] Step 5:

[0416] Based on the analysis results, the server generates scripts to automate identified routines. For example, if a user opens the same file at the same time every day, it will create a script to automate that operation.

[0417] Step 6:

[0418] The server notifies the user of the generated scripts and suggested business improvements. At this stage, the user can review the proposed automations and approve or reject them.

[0419] Step 7:

[0420] If the user approves the script, the automation is executed by running the script on the device. This frees the user from monotonous tasks.

[0421] Step 8:

[0422] The server monitors the progress of tasks across the entire organization in real time. Through the task management dashboard, it visualizes the progress and resource usage of each member and automatically issues alerts if any anomalies are detected.

[0423] Step 9:

[0424] Users and teams make adjustments to improve operational efficiency based on feedback and suggestions from the server. This, in turn, increases the productivity of the entire organization.

[0425] (Example 1)

[0426] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0427] As the need for business efficiency and automation increases, it is essential to appropriately identify and automate repetitive tasks in the daily work activities of users. Furthermore, optimizing operations by understanding the overall progress of operations within the organization and quickly detecting anomalies is a challenge. Additionally, it is necessary to generate suggestions for continuous business improvement by utilizing past business data.

[0428] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0429] In this invention, the server includes means for monitoring user operations and collecting information, means for using computational models to analyze the data, and means for generating instructions to automate repetitive tasks. This enables increased efficiency and automation of operations. Furthermore, by monitoring the progress of the organization's operations and providing warnings and work improvement suggestions based on anomaly detection, it becomes possible to optimize operations and achieve continuous improvement.

[0430] "Monitoring user operations" means recording inputs and actions on a terminal in order to understand the work activities and intentions of the user.

[0431] "Collecting information" means gathering and organizing data for a specific purpose and using it for subsequent analysis and processing.

[0432] A "computational model" is a program based on mathematical or statistical methods used for data analysis, and is a model used for pattern extraction and prediction.

[0433] "Business activities" generally refer to a series of actions and processes performed in the course of carrying out business.

[0434] "Automating" means minimizing human intervention and having software or machines perform specific tasks.

[0435] "Monitoring work progress" means observing the progress of work to keep track of its status and confirming that it is proceeding according to plan.

[0436] "Anomaly detection" refers to identifying data or situations that deviate from the normal range and pointing out areas that require attention.

[0437] "Warning" refers to providing notifications or alerts to draw the user's attention.

[0438] A "work improvement proposal" is a suggestion for reforms or specific methods to improve efficiency in business operations.

[0439] This invention is a system that achieves automation and efficiency in business operations, and is built through the collaboration of terminals and servers. Specific embodiments are shown below.

[0440] The terminal has a client program installed to monitor user actions and collect information. This program records user work-related actions in real time and collects data on location and applications used. The collected data is sent to the server in an appropriate format.

[0441] The server is responsible for analyzing the received data. The processes running on the server include computational models for data analysis based on user actions (for example, machine learning models built in a Python environment). These models extract patterns of business activities from the collected data and identify recurring business activities. Based on these results, automation instructions are generated. These instructions are then provided to the user from the server, enabling the execution of automated business processes.

[0442] As a concrete example, the server generates a script to automatically open files that a user should open at a specific time each day. The user receives this instruction and can streamline their work through the automated process. Furthermore, the server can leverage historical data to suggest new business improvements, contributing to continuous operational efficiency.

[0443] Generative AI models are highly effective in recognizing patterns in such business activities and generating automation instructions. An example of a prompt is: "Propose an algorithm that analyzes the applications and operation patterns used by the user daily, and based on that, generates automation instructions aimed at improving work efficiency." This prompt serves as a guideline for accurately analyzing the user's intent and maximizing the potential of automation.

[0444] This system aims to speed up operations, improve accuracy, and enhance convenience and efficiency for users.

[0445] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0446] Step 1:

[0447] The terminal monitors user operations in real time and collects data on work-related activities. Specifically, it collects mouse clicks, keyboard input, application usage logs, and location information. This data is recorded as a time-series operation log. The collected data is processed into a predefined format by the client program and prepared for transmission to the server.

[0448] Step 2:

[0449] The terminal periodically sends collected operation log data to the server. This data is encrypted to ensure secure transfer. The transmitted data includes the user ID, timestamp, and details of each action. This data is stored on the server as raw material for analysis.

[0450] Step 3:

[0451] The server receives the incoming data and analyzes it using a machine learning computational model. Specifically, it uses the Python pandas library to format the data and clean up outliers. Next, it extracts patterns of specific business activities through analysis using scikit-learn. This analysis allows for the identification of recurring business activities.

[0452] Step 4:

[0453] The server generates automation instructions based on patterns derived from the analysis. For example, it might generate a script to automatically open files that need to be opened at a specific time each day, or a template for sending standard emails. These instructions are generated as Python or JavaScript scripts and ready to be provided to the user.

[0454] Step 5:

[0455] The user receives and executes automation instructions sent from the server. In this process, the user runs a provided script, automating a portion of their work. This automated action allows the user to reduce the effort required for routine daily tasks.

[0456] Step 6:

[0457] The server monitors the overall operational progress of the organization. Data is visualized based on task progress, and delays are detected by an anomaly detection algorithm. In response to detected anomalies, the server sends alerts to users via email or notifications to support more efficient work execution.

[0458] Step 7:

[0459] The server analyzes past business data and proposes business improvement measures. For example, it analyzes the steps users took to perform tasks and generates suggestions for more efficient methods or the introduction of new tools. These suggestions are notified to users and incorporated into business processes.

[0460] (Application Example 1)

[0461] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0462] There is a need for a system that improves the operational efficiency of work equipment and robots used in factories, enabling automation and progress monitoring of tasks, and allowing for immediate response in the event of an anomaly. This will facilitate collaborative work with workers and improve productivity.

[0463] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0464] In this invention, the server includes means for monitoring and recording user operations, means for visualizing the progress of tasks based on those operations, and means for generating optimization instructions and displaying them on a visual device. This makes it possible to automatically improve the operational efficiency of work equipment within a factory and immediately improve operations.

[0465] "Means for monitoring and recording user operations" refers to technology that allows a system to check in real time the operation and behavior of equipment by a specific user and save that information as a history.

[0466] "Methods for visualizing work progress" refer to technologies that display the progress of work in an easily understandable format, enabling users and managers to quickly grasp the degree of progress and any delays in the work.

[0467] "Means for generating optimization instructions and displaying them on a visual device" refers to a technology that analyzes efficient operating methods in equipment and work environments, and provides a function to encourage improved operation by visually presenting the results to the user.

[0468] To implement this invention, a system is constructed that links a server and a terminal. The terminal has a program installed to monitor and record user operations, collecting daily work and equipment operation information in real time and sending the data to the server. This program continuously tracks the actions performed by the user and stores them as a history.

[0469] The server uses Python and Scikit-learn to analyze received data and runs a program to visualize the progress of tasks. This allows for immediate warnings if any anomalies are detected or if the work is progressing as planned. The server also generates optimal operational instructions and displays them on a terminal or visual device (e.g., Android smart glasses) using Unity. Through this visual device, the user is presented with visualized instructions outlining specific work procedures.

[0470] As a concrete example, in parts maintenance work within a factory, the inspection process of parts, which is normally performed manually, can be significantly streamlined by using efficient process instructions automatically generated from the robot's motion patterns. Users can improve the accuracy and speed of their work by relying on visual information displayed on smart glasses.

[0471] This allows us to prompt the generated AI model and utilize instruction statements such as, "Develop an application that monitors the operation of factory robots in real time, suggests optimal actions to improve efficiency, and visually presents specific instructions."

[0472] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0473] Step 1:

[0474] The terminal monitors user actions and collects action data in real time. Input is user action information, and output is a sequential history of those actions. The terminal program records the action details and prepares them for transmission to the server.

[0475] Step 2:

[0476] The server receives operation data sent from the terminal and stores it in a database. The input is the operation data from the terminal, and the output is the record stored in the database. By storing it in the database, the server efficiently manages the operation history.

[0477] Step 3:

[0478] The server analyzes stored data to detect operational trends and anomalies. The input is the operation history read from the database, and the output is trend analysis and anomaly warning information as a result of the analysis. It uses Scikit-learn to model the data and identify outliers and patterns.

[0479] Step 4:

[0480] The server generates optimized operation instructions and sends them to the visual device. The input is the analysis result, and the output is the visual operation instructions. Based on the analysis result, steps to optimize the process are determined, and instructions are generated to be displayed on the smart glasses via Unity.

[0481] Step 5:

[0482] The user performs tasks by following instructions from a visual device. The input is the display instructions from the visual device, and the output is the result of the performed task. The user sees the steps presented on the smart glasses and performs the tasks specifically, efficiently advancing their work.

[0483] Step 6:

[0484] The server receives the work results and feeds that information back into the next analysis. The input is the work results performed by the user, and the output is improvement data for the next analysis. Based on this feedback, the server provides prompts to the generating AI model to further improve efficiency.

[0485] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0486] The present invention is implemented in a form that combines emotion recognition functionality with a system that supports the automation and efficiency of business operations. The system consists of a terminal, a server, and an emotion engine.

[0487] The terminal has a client program installed that monitors and records the user's operation history and application usage. The terminal is also equipped with a camera and microphone, through which an emotion engine recognizes the user's emotional state in real time. The camera analyzes facial expressions using facial recognition technology, and the microphone performs voice analysis.

[0488] User operation data and emotional data are sent from the terminal to the server. The server receives this data and analyzes the user's usage patterns and emotional state. The emotional data recognized by the emotion engine is analyzed using machine learning algorithms to detect changes in the user's stress level and motivation.

[0489] Based on the analysis results, the server generates scripts to automate repetitive tasks and suggestions to improve work efficiency. In addition, it has a function to suggest task priorities and notify users of the need for breaks, taking into account the impact of the user's emotional state on work efficiency.

[0490] For example, if the emotion engine detects that a user is experiencing increased stress while performing a particular task, the server will generate an automated script for that task and suggest automating it in the future. Furthermore, it will improve the work environment by sending a notification to the user recommending that they pause the task and take a short break.

[0491] This system allows users to optimize their work environment in a way that suits their individual needs, thereby reducing physical and mental strain while improving productivity.

[0492] The following describes the processing flow.

[0493] Step 1:

[0494] The device monitors user activity and records input data, application usage, and information on opened and closed files. It also uses its built-in camera and microphone to analyze user facial expressions and voice tone, collecting emotional data.

[0495] Step 2:

[0496] The device encrypts the collected operational and emotional data and periodically sends it to the server. This ensures that data is processed while protecting user privacy.

[0497] Step 3:

[0498] The server stores the received data and analyzes it using machine learning algorithms. The purpose of the analysis is to identify patterns in user usage, recurring work processes, and emotional states.

[0499] Step 4:

[0500] The server generates scripts for business processes that can be automated based on the analysis results. At the same time, it considers the user's emotional state and generates suggestions for appropriate breaks and tasks if signs of stress or fatigue are detected.

[0501] Step 5:

[0502] The server notifies the user of the generated automation scripts and suggestions. The user can review them and choose whether to accept the automation. They can also make a similar decision regarding the suggested breaks.

[0503] Step 6:

[0504] If a user approves the automation, a script is executed on the terminal, automating the specified tasks. This frees users from tedious administrative work and improves work efficiency.

[0505] Step 7:

[0506] The server continuously monitors the overall progress of operations and evaluates the workload of the entire organization. If an anomaly is detected, it automatically sends an alert and notifies the administrator.

[0507] Step 8:

[0508] Users and administrators can use alerts and suggestions from the server to improve business processes and optimize resources. This is expected to lead to sustained productivity improvements.

[0509] (Example 2)

[0510] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0511] In today's work environment, users are required to efficiently handle diverse tasks and improve work efficiency while reducing physical and mental burden. However, conventional automation systems often struggle to dynamically optimize work efficiency while considering the user's emotional state, or to suggest optimal break times, thus failing to maximize overall work productivity.

[0512] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0513] In this invention, the server includes means for monitoring the user's operation history and acquiring usage trends, means for identifying recurring tasks based on usage trends and emotional states, and means for analyzing the user's emotional state in real time using emotion recognition technology. This enables the automation of dynamic tasks, adjustment of efficiency priorities, and suggestion of optimal break times, all while considering the user's emotional state.

[0514] "User operation history" refers to a record of operations performed by the user on the device, including data such as which applications were used and to what extent.

[0515] "Usage patterns" refer to patterns in how and how often users use a particular application or feature.

[0516] "Emotional state" refers to psychological elements such as stress levels and motivation exhibited by users during work, and is acquired through data analysis using cameras and microphones.

[0517] "Repetitive tasks" refer to routine tasks and routine work that users perform frequently, and are therefore suitable for automation.

[0518] "Adjusting work priorities" refers to the process of rearranging tasks that need to be done based on the user's emotional state and work situation, thereby supporting efficient work execution.

[0519] "Break timing suggestions" refers to a function that notifies users of the optimal timing for taking a break, taking into account their physical and mental burden.

[0520] "Emotion recognition technology" refers to technology that uses cameras and microphones to analyze facial expressions and voices in real time and infer the emotional state of the user.

[0521] This invention is a system that supports the automation and efficiency of business operations, and consists of a terminal, a server, and an emotion engine. Specifically, the terminal holds a client program for acquiring the user's operation history and operates the emotion engine through a camera and microphone. The camera analyzes the user's facial expressions using facial recognition technology, and the microphone performs voice analysis. Through these devices, the user's emotional state can be recognized in real time.

[0522] Operational and emotional data transmitted from the device are aggregated on a server. The server analyzes this data to identify the user's usage patterns and emotional state. In particular, emotional data obtained by the emotion engine is processed using machine learning algorithms to help detect changes in the user's stress levels and motivation.

[0523] Based on the analysis results, the server generates automation scripts and suggestions to help users streamline their work. For example, if the emotion engine determines that a user is stressed by a particular task, it will suggest a script to automate that task. It can also notify the user of appropriate break times, thereby improving the work environment.

[0524] As a concrete example, if a user detects high stress while creating a document, the server will send a notification such as, "We suggest automating the task using a document template. A 5-minute break is recommended." A concrete example of a prompt message for the generating AI model would be, "If the stress level rises, generate an automation script for the task and suggest a break."

[0525] This approach provides concrete implementation methods for optimizing the user's work environment, reducing physical and mental burden, and increasing productivity.

[0526] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0527] Step 1:

[0528] The device monitors the user's operation history and records usage. Specifically, it receives input such as the name of the application the user is using, the order of operations, and the duration of those operations, and stores this data in a database. This provides basic data for understanding user usage trends.

[0529] Step 2:

[0530] Using a camera and microphone attached to the device, the emotion engine recognizes the user's emotional state in real time. The camera acquires feature points of the user's face as input and analyzes facial expressions using facial recognition technology. The microphone collects voice input and analyzes voice tone and speaking speed. Based on these analysis results, emotional state data is output.

[0531] Step 3:

[0532] The terminal sends the acquired operation history data and emotional state data to the server. Here, the data is formatted and encrypted before being output as transmitted data.

[0533] Step 4:

[0534] The server analyzes the received operation history data and emotional state data. Based on the input data, it utilizes machine learning models to identify user usage patterns and changes in emotional state. It performs data calculations and outputs analysis results showing each user's stress level and changes in motivation.

[0535] Step 5:

[0536] Based on the analysis results, the server generates suggestions to improve the user's work efficiency. It creates automation scripts for repetitive tasks and suggests breaks or more efficient work processes for stressful tasks. These suggestions are output as prompts and notified to the user.

[0537] Step 6:

[0538] The user receives a notification and decides whether to accept the suggested automation script. This input is returned to the terminal, which updates the work procedure. This process ultimately improves the user's work environment.

[0539] (Application Example 2)

[0540] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0541] Many tasks and household chores that users face daily at home and in the workplace are frequently repeated, and there is a need for increased efficiency. Furthermore, because work methods that take into account the emotional state of users are not readily available, these tasks can lead to significant mental and physical burden and decreased work efficiency. Therefore, it is desirable to appropriately support and streamline work and household tasks in accordance with the emotional state of users, thereby reducing their mental and physical burden.

[0542] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0543] In this invention, the server includes means for acquiring user usage patterns, means for generating instructions to automate repetitive administrative tasks, and means for recognizing the user's emotional state in real time and proposing support activities based on that state. This enables efficient and effective work support tailored to the individual user's condition.

[0544] "Means for acquiring user usage trends" refers to a device or method that records the operations and selections made by a user and collects those patterns as data.

[0545] "Means for identifying repetitive administrative tasks" refers to a device or method that analyzes the aforementioned usage trend data and identifies routine tasks performed on a daily basis.

[0546] "Means for generating instructions to automate clerical tasks" refers to a device or method for constructing specific processes or procedures for efficiently automating identified repetitive tasks.

[0547] "Means for recognizing a user's emotional state in real time" refers to a device or method that uses a camera and microphone to analyze a user's facial expressions and voice to understand their emotions and grasp their psychological state at any given time.

[0548] "Means of proposing support activities" refers to a device or method that presents the most appropriate actions or services based on the user's emotional state and provides support that meets the user's needs.

[0549] In implementing this invention, the system mainly consists of a terminal, a server, an emotion engine, and a home robot.

[0550] The device has a client program installed to acquire the user's usage patterns. The device is equipped with a camera and a microphone, and the emotion engine recognizes the user's emotional state in real time through these devices. The camera is used to analyze facial expressions using facial recognition technology, and the microphone is used for voice analysis.

[0551] User operation data and emotional data are sent from the terminal to the server. The server receives this data and analyzes the user's usage patterns and emotional state. In particular, emotional data recognized by the emotion engine is analyzed using machine learning algorithms to detect changes in the user's stress level and motivation. Based on the user's emotional state and usage patterns, the server suggests support activities for work and household chores.

[0552] Home robots support the daily lives of families through physical assistance and conversation, based on instructions for support activities sent from a server. For example, if someone in the family is experiencing stress, the robot can analyze their condition and play relaxation music or suggest taking a break.

[0553] For example, if the server detects increased stress levels in a user while they are performing a task, it can suggest automated task processing to reduce stress in the future. Also, if a child feels stressed while doing homework, the robot might suggest, "Why don't you take a short break? Shall I get you a drink?"

[0554] An example of a prompt is, "Please tell me how to analyze the emotional state of family members and suggest support activities within the home as needed."

[0555] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0556] Step 1:

[0557] The device uses its camera and microphone to capture the user's facial expressions and voice in real time. The input consists of camera video and audio data. Facial expressions are analyzed from the camera video using OpenCV's facial recognition technology, and the audio data is analyzed using the Google Speech-to-Text API. The output consists of analyzed facial expression data and audio-to-text data.

[0558] Step 2:

[0559] The device analyzes facial expression data and voice-to-text data and sends it to the server. The data is sent to the cloud server via a secure communication protocol. The input is facial expression data and voice-to-text data, and this data is sent directly to the server as output.

[0560] Step 3:

[0561] The server analyzes the facial expression data and voice / text data it receives and uses a machine learning algorithm to evaluate the user's emotional state. This analysis determines the degree of stress and motivation the user is experiencing. The input is facial expression data and voice / text data, and the output is evaluation data indicating the emotional state.

[0562] Step 4:

[0563] The server generates appropriate support activities based on the user's emotional state. It determines the actions necessary for the user's stress relief and efficient task completion. The input is emotional state evaluation data, and the output is instruction data indicating the content of the support activity.

[0564] Step 5:

[0565] The server generates instruction data for support activities and sends it to the home robot, which then carries it out. This is done through seamless communication over the network, with the robot interacting with the user through voice and actions. The input is the instruction data sent from the server, and the output is the specific support actions performed by the robot within the home.

[0566] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0567] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0568] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0569] [Fourth Embodiment]

[0570] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0571] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0572] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0573] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0574] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0575] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0576] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0577] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0578] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0579] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0580] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0581] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0582] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0583] This invention provides a system for achieving automation and efficiency improvements in business operations. The system consists of a terminal and a server working in conjunction.

[0584] The terminal has a client program installed to monitor and record user actions. This program collects real-time information on user actions, application usage, and location during daily work, and sends the data to a server.

[0585] The server receives data sent from the terminal and runs a program to analyze user usage patterns. Specifically, it uses machine learning algorithms to identify repetitive office tasks and movement patterns. Based on this analysis, the server generates instructions to automate repetitive tasks.

[0586] Meanwhile, the server also runs a program that monitors the organization's work progress. This program visualizes the progress of tasks and has the function of detecting anomalies. When an anomaly is detected, the server sends an alert to the user and suggests necessary improvements.

[0587] The user receives instructions from the server and executes the automation. For example, the server can generate a script that automatically opens files the user should open daily and notify the user, allowing the user to work more efficiently. The server can also analyze past data and make new suggestions for improving work processes.

[0588] Through this series of operations, the present invention achieves the automation of business processes and improves operational efficiency for users.

[0589] The following describes the processing flow.

[0590] Step 1:

[0591] The device records the user's activity history in real time. This includes keyboard and mouse input, application launches and shutdowns, usage time, and document opening and closing. It also obtains movement history using the smartphone's location services.

[0592] Step 2:

[0593] The device encrypts the data collected at regular intervals and sends it to the server. Data transmission is performed using a secure protocol.

[0594] Step 3:

[0595] The server stores the received data and stores it in a database. The data is organized by user and prepared for future analysis.

[0596] Step 4:

[0597] The server analyzes the accumulated data using machine learning algorithms. This analysis identifies repetitive operations frequently performed by users and routines carried out for specific purposes.

[0598] Step 5:

[0599] Based on the analysis results, the server generates scripts to automate identified routines. For example, if a user opens the same file at the same time every day, it will create a script to automate that operation.

[0600] Step 6:

[0601] The server notifies the user of the generated scripts and suggested business improvements. At this stage, the user can review the proposed automations and approve or reject them.

[0602] Step 7:

[0603] If the user approves the script, the automation is executed by running the script on the device. This frees the user from monotonous tasks.

[0604] Step 8:

[0605] The server monitors the progress of tasks across the entire organization in real time. Through the task management dashboard, it visualizes the progress and resource usage of each member and automatically issues alerts if any anomalies are detected.

[0606] Step 9:

[0607] Users and teams make adjustments to improve operational efficiency based on feedback and suggestions from the server. This, in turn, increases the productivity of the entire organization.

[0608] (Example 1)

[0609] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0610] As the need for business efficiency and automation increases, it is essential to appropriately identify and automate repetitive tasks in the daily work activities of users. Furthermore, optimizing operations by understanding the overall progress of operations within the organization and quickly detecting anomalies is a challenge. Additionally, it is necessary to generate suggestions for continuous business improvement by utilizing past business data.

[0611] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0612] In this invention, the server includes means for monitoring user operations and collecting information, means for using computational models to analyze the data, and means for generating instructions to automate repetitive tasks. This enables increased efficiency and automation of operations. Furthermore, by monitoring the progress of the organization's operations and providing warnings and work improvement suggestions based on anomaly detection, it becomes possible to optimize operations and achieve continuous improvement.

[0613] "Monitoring user operations" means recording inputs and actions on a terminal in order to understand the work activities and intentions of the user.

[0614] "Collecting information" means gathering and organizing data for a specific purpose and using it for subsequent analysis and processing.

[0615] A "computational model" is a program based on mathematical or statistical methods used for data analysis, and is a model used for pattern extraction and prediction.

[0616] "Business activities" generally refer to a series of actions and processes performed in the course of carrying out business.

[0617] "Automating" means minimizing human intervention and having software or machines perform specific tasks.

[0618] "Monitoring work progress" means observing the progress of work to keep track of its status and confirming that it is proceeding according to plan.

[0619] "Anomaly detection" refers to identifying data or situations that deviate from the normal range and pointing out areas that require attention.

[0620] "Warning" refers to providing notifications or alerts to draw the user's attention.

[0621] A "work improvement proposal" is a suggestion for reforms or specific methods to improve efficiency in business operations.

[0622] This invention is a system that achieves automation and efficiency in business operations, and is built through the collaboration of terminals and servers. Specific embodiments are shown below.

[0623] The terminal has a client program installed to monitor user actions and collect information. This program records user work-related actions in real time and collects data on location and applications used. The collected data is sent to the server in an appropriate format.

[0624] The server is responsible for analyzing the received data. The processes running on the server include computational models for data analysis based on user actions (for example, machine learning models built in a Python environment). These models extract patterns of business activities from the collected data and identify recurring business activities. Based on these results, automation instructions are generated. These instructions are then provided to the user from the server, enabling the execution of automated business processes.

[0625] As a concrete example, the server generates a script to automatically open files that a user should open at a specific time each day. The user receives this instruction and can streamline their work through the automated process. Furthermore, the server can leverage historical data to suggest new business improvements, contributing to continuous operational efficiency.

[0626] Generative AI models are highly effective in recognizing patterns in such business activities and generating automation instructions. An example of a prompt is: "Propose an algorithm that analyzes the applications and operation patterns used by the user daily, and based on that, generates automation instructions aimed at improving work efficiency." This prompt serves as a guideline for accurately analyzing the user's intent and maximizing the potential of automation.

[0627] This system aims to speed up operations, improve accuracy, and enhance convenience and efficiency for users.

[0628] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0629] Step 1:

[0630] The terminal monitors user operations in real time and collects data on work-related activities. Specifically, it collects mouse clicks, keyboard input, application usage logs, and location information. This data is recorded as a time-series operation log. The collected data is processed into a predefined format by the client program and prepared for transmission to the server.

[0631] Step 2:

[0632] The terminal periodically sends collected operation log data to the server. This data is encrypted to ensure secure transfer. The transmitted data includes the user ID, timestamp, and details of each action. This data is stored on the server as raw material for analysis.

[0633] Step 3:

[0634] The server receives the incoming data and analyzes it using a machine learning computational model. Specifically, it uses the Python pandas library to format the data and clean up outliers. Next, it extracts patterns of specific business activities through analysis using scikit-learn. This analysis allows for the identification of recurring business activities.

[0635] Step 4:

[0636] The server generates automation instructions based on patterns derived from the analysis. For example, it might generate a script to automatically open files that need to be opened at a specific time each day, or a template for sending standard emails. These instructions are generated as Python or JavaScript scripts and ready to be provided to the user.

[0637] Step 5:

[0638] The user receives and executes automation instructions sent from the server. In this process, the user runs a provided script, automating a portion of their work. This automated action allows the user to reduce the effort required for routine daily tasks.

[0639] Step 6:

[0640] The server monitors the overall operational progress of the organization. Data is visualized based on task progress, and delays are detected by an anomaly detection algorithm. In response to detected anomalies, the server sends alerts to users via email or notifications to support more efficient work execution.

[0641] Step 7:

[0642] The server analyzes past business data and proposes business improvement measures. For example, it analyzes the steps users took to perform tasks and generates suggestions for more efficient methods or the introduction of new tools. These suggestions are notified to users and incorporated into business processes.

[0643] (Application Example 1)

[0644] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0645] There is a need for a system that improves the operational efficiency of work equipment and robots used in factories, enabling automation and progress monitoring of tasks, and allowing for immediate response in the event of an anomaly. This will facilitate collaborative work with workers and improve productivity.

[0646] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0647] In this invention, the server includes means for monitoring and recording user operations, means for visualizing the progress of tasks based on those operations, and means for generating optimization instructions and displaying them on a visual device. This makes it possible to automatically improve the operational efficiency of work equipment within a factory and immediately improve operations.

[0648] "Means for monitoring and recording user operations" refers to technology that allows a system to check in real time the operation and behavior of equipment by a specific user and save that information as a history.

[0649] "Methods for visualizing work progress" refer to technologies that display the progress of work in an easily understandable format, enabling users and managers to quickly grasp the degree of progress and any delays in the work.

[0650] "Means for generating optimization instructions and displaying them on a visual device" refers to a technology that analyzes efficient operating methods in equipment and work environments, and provides a function to encourage improved operation by visually presenting the results to the user.

[0651] To implement this invention, a system is constructed that links a server and a terminal. The terminal has a program installed to monitor and record user operations, collecting daily work and equipment operation information in real time and sending the data to the server. This program continuously tracks the actions performed by the user and stores them as a history.

[0652] The server uses Python and Scikit-learn to analyze received data and runs a program to visualize the progress of tasks. This allows for immediate warnings if any anomalies are detected or if the work is progressing as planned. The server also generates optimal operational instructions and displays them on a terminal or visual device (e.g., Android smart glasses) using Unity. Through this visual device, the user is presented with visualized instructions outlining specific work procedures.

[0653] As a concrete example, in parts maintenance work within a factory, the inspection process of parts, which is normally performed manually, can be significantly streamlined by using efficient process instructions automatically generated from the robot's motion patterns. Users can improve the accuracy and speed of their work by relying on visual information displayed on smart glasses.

[0654] This allows us to prompt the generated AI model and utilize instruction statements such as, "Develop an application that monitors the operation of factory robots in real time, suggests optimal actions to improve efficiency, and visually presents specific instructions."

[0655] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0656] Step 1:

[0657] The terminal monitors user actions and collects action data in real time. Input is user action information, and output is a sequential history of those actions. The terminal program records the action details and prepares them for transmission to the server.

[0658] Step 2:

[0659] The server receives operation data sent from the terminal and stores it in a database. The input is the operation data from the terminal, and the output is the record stored in the database. By storing it in the database, the server efficiently manages the operation history.

[0660] Step 3:

[0661] The server analyzes stored data to detect operational trends and anomalies. The input is the operation history read from the database, and the output is trend analysis and anomaly warning information as a result of the analysis. It uses Scikit-learn to model the data and identify outliers and patterns.

[0662] Step 4:

[0663] The server generates optimized operation instructions and sends them to the visual device. The input is the analysis result, and the output is the visual operation instructions. Based on the analysis result, steps to optimize the process are determined, and instructions are generated to be displayed on the smart glasses via Unity.

[0664] Step 5:

[0665] The user performs tasks by following instructions from a visual device. The input is the display instructions from the visual device, and the output is the result of the performed task. The user sees the steps presented on the smart glasses and performs the tasks specifically, efficiently advancing their work.

[0666] Step 6:

[0667] The server receives the work results and feeds that information back into the next analysis. The input is the work results performed by the user, and the output is improvement data for the next analysis. Based on this feedback, the server provides prompts to the generating AI model to further improve efficiency.

[0668] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0669] The present invention is implemented in a form that combines emotion recognition functionality with a system that supports the automation and efficiency of business operations. The system consists of a terminal, a server, and an emotion engine.

[0670] The terminal has a client program installed that monitors and records the user's operation history and application usage. The terminal is also equipped with a camera and microphone, through which an emotion engine recognizes the user's emotional state in real time. The camera analyzes facial expressions using facial recognition technology, and the microphone performs voice analysis.

[0671] User operation data and emotional data are sent from the terminal to the server. The server receives this data and analyzes the user's usage patterns and emotional state. The emotional data recognized by the emotion engine is analyzed using machine learning algorithms to detect changes in the user's stress level and motivation.

[0672] Based on the analysis results, the server generates scripts to automate repetitive tasks and suggestions to improve work efficiency. In addition, it has a function to suggest task priorities and notify users of the need for breaks, taking into account the impact of the user's emotional state on work efficiency.

[0673] For example, if the emotion engine detects that a user is experiencing increased stress while performing a particular task, the server will generate an automated script for that task and suggest automating it in the future. Furthermore, it will improve the work environment by sending a notification to the user recommending that they pause the task and take a short break.

[0674] This system allows users to optimize their work environment in a way that suits their individual needs, thereby reducing physical and mental strain while improving productivity.

[0675] The following describes the processing flow.

[0676] Step 1:

[0677] The device monitors user activity and records input data, application usage, and information on opened and closed files. It also uses its built-in camera and microphone to analyze user facial expressions and voice tone, collecting emotional data.

[0678] Step 2:

[0679] The device encrypts the collected operational and emotional data and periodically sends it to the server. This ensures that data is processed while protecting user privacy.

[0680] Step 3:

[0681] The server stores the received data and analyzes it using machine learning algorithms. The purpose of the analysis is to identify patterns in user usage, recurring work processes, and emotional states.

[0682] Step 4:

[0683] The server generates scripts for business processes that can be automated based on the analysis results. At the same time, it considers the user's emotional state and generates suggestions for appropriate breaks and tasks if signs of stress or fatigue are detected.

[0684] Step 5:

[0685] The server notifies the user of the generated automation scripts and suggestions. The user can review them and choose whether to accept the automation. They can also make a similar decision regarding the suggested breaks.

[0686] Step 6:

[0687] If a user approves the automation, a script is executed on the terminal, automating the specified tasks. This frees users from tedious administrative work and improves work efficiency.

[0688] Step 7:

[0689] The server continuously monitors the overall progress of operations and evaluates the workload of the entire organization. If an anomaly is detected, it automatically sends an alert and notifies the administrator.

[0690] Step 8:

[0691] Users and administrators can use alerts and suggestions from the server to improve business processes and optimize resources. This is expected to lead to sustained productivity improvements.

[0692] (Example 2)

[0693] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0694] In today's work environment, users are required to efficiently handle diverse tasks and improve work efficiency while reducing physical and mental burden. However, conventional automation systems often struggle to dynamically optimize work efficiency while considering the user's emotional state, or to suggest optimal break times, thus failing to maximize overall work productivity.

[0695] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0696] In this invention, the server includes means for monitoring the user's operation history and acquiring usage trends, means for identifying recurring tasks based on usage trends and emotional states, and means for analyzing the user's emotional state in real time using emotion recognition technology. This enables the automation of dynamic tasks, adjustment of efficiency priorities, and suggestion of optimal break times, all while considering the user's emotional state.

[0697] "User operation history" refers to a record of operations performed by the user on the device, including data such as which applications were used and to what extent.

[0698] "Usage patterns" refer to patterns in how and how often users use a particular application or feature.

[0699] "Emotional state" refers to psychological elements such as stress levels and motivation exhibited by users during work, and is acquired through data analysis using cameras and microphones.

[0700] "Repetitive tasks" refer to routine tasks and routine work that users perform frequently, and are therefore suitable for automation.

[0701] "Adjusting work priorities" refers to the process of rearranging tasks that need to be done based on the user's emotional state and work situation, thereby supporting efficient work execution.

[0702] "Break timing suggestions" refers to a function that notifies users of the optimal timing for taking a break, taking into account their physical and mental burden.

[0703] "Emotion recognition technology" refers to technology that uses cameras and microphones to analyze facial expressions and voices in real time and infer the emotional state of the user.

[0704] This invention is a system that supports the automation and efficiency of business operations, and consists of a terminal, a server, and an emotion engine. Specifically, the terminal holds a client program for acquiring the user's operation history and operates the emotion engine through a camera and microphone. The camera analyzes the user's facial expressions using facial recognition technology, and the microphone performs voice analysis. Through these devices, the user's emotional state can be recognized in real time.

[0705] Operational and emotional data transmitted from the device are aggregated on a server. The server analyzes this data to identify the user's usage patterns and emotional state. In particular, emotional data obtained by the emotion engine is processed using machine learning algorithms to help detect changes in the user's stress levels and motivation.

[0706] Based on the analysis results, the server generates automation scripts and suggestions to help users streamline their work. For example, if the emotion engine determines that a user is stressed by a particular task, it will suggest a script to automate that task. It can also notify the user of appropriate break times, thereby improving the work environment.

[0707] As a concrete example, if a user detects high stress while creating a document, the server will send a notification such as, "We suggest automating the task using a document template. A 5-minute break is recommended." A concrete example of a prompt message for the generating AI model would be, "If the stress level rises, generate an automation script for the task and suggest a break."

[0708] This approach provides concrete implementation methods for optimizing the user's work environment, reducing physical and mental burden, and increasing productivity.

[0709] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0710] Step 1:

[0711] The device monitors the user's operation history and records usage. Specifically, it receives input such as the name of the application the user is using, the order of operations, and the duration of those operations, and stores this data in a database. This provides basic data for understanding user usage trends.

[0712] Step 2:

[0713] Using a camera and microphone attached to the device, the emotion engine recognizes the user's emotional state in real time. The camera acquires feature points of the user's face as input and analyzes facial expressions using facial recognition technology. The microphone collects voice input and analyzes voice tone and speaking speed. Based on these analysis results, emotional state data is output.

[0714] Step 3:

[0715] The terminal sends the acquired operation history data and emotional state data to the server. Here, the data is formatted and encrypted before being output as transmitted data.

[0716] Step 4:

[0717] The server analyzes the received operation history data and emotional state data. Based on the input data, it utilizes machine learning models to identify user usage patterns and changes in emotional state. It performs data calculations and outputs analysis results showing each user's stress level and changes in motivation.

[0718] Step 5:

[0719] Based on the analysis results, the server generates suggestions to improve the user's work efficiency. It creates automation scripts for repetitive tasks and suggests breaks or more efficient work processes for stressful tasks. These suggestions are output as prompts and notified to the user.

[0720] Step 6:

[0721] The user receives a notification and decides whether to accept the suggested automation script. This input is returned to the terminal, which updates the work procedure. This process ultimately improves the user's work environment.

[0722] (Application Example 2)

[0723] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0724] Many tasks and household chores that users face daily at home and in the workplace are frequently repeated, and there is a need for increased efficiency. Furthermore, because work methods that take into account the emotional state of users are not readily available, these tasks can lead to significant mental and physical burden and decreased work efficiency. Therefore, it is desirable to appropriately support and streamline work and household tasks in accordance with the emotional state of users, thereby reducing their mental and physical burden.

[0725] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0726] In this invention, the server includes means for acquiring user usage patterns, means for generating instructions to automate repetitive administrative tasks, and means for recognizing the user's emotional state in real time and proposing support activities based on that state. This enables efficient and effective work support tailored to the individual user's condition.

[0727] "Means for acquiring user usage trends" refers to a device or method that records the operations and selections made by a user and collects those patterns as data.

[0728] "Means for identifying repetitive administrative tasks" refers to a device or method that analyzes the aforementioned usage trend data and identifies routine tasks performed on a daily basis.

[0729] "Means for generating instructions to automate clerical tasks" refers to a device or method for constructing specific processes or procedures for efficiently automating identified repetitive tasks.

[0730] "Means for recognizing a user's emotional state in real time" refers to a device or method that uses a camera and microphone to analyze a user's facial expressions and voice to understand their emotions and grasp their psychological state at any given time.

[0731] "Means of proposing support activities" refers to a device or method that presents the most appropriate actions or services based on the user's emotional state and provides support that meets the user's needs.

[0732] In implementing this invention, the system mainly consists of a terminal, a server, an emotion engine, and a home robot.

[0733] The device has a client program installed to acquire the user's usage patterns. The device is equipped with a camera and a microphone, and the emotion engine recognizes the user's emotional state in real time through these devices. The camera is used to analyze facial expressions using facial recognition technology, and the microphone is used for voice analysis.

[0734] User operation data and emotional data are sent from the terminal to the server. The server receives this data and analyzes the user's usage patterns and emotional state. In particular, emotional data recognized by the emotion engine is analyzed using machine learning algorithms to detect changes in the user's stress level and motivation. Based on the user's emotional state and usage patterns, the server suggests support activities for work and household chores.

[0735] Home robots support the daily lives of families through physical assistance and conversation, based on instructions for support activities sent from a server. For example, if someone in the family is experiencing stress, the robot can analyze their condition and play relaxation music or suggest taking a break.

[0736] For example, if the server detects increased stress levels in a user while they are performing a task, it can suggest automated task processing to reduce stress in the future. Also, if a child feels stressed while doing homework, the robot might suggest, "Why don't you take a short break? Shall I get you a drink?"

[0737] An example of a prompt is, "Please tell me how to analyze the emotional state of family members and suggest support activities within the home as needed."

[0738] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0739] Step 1:

[0740] The device uses its camera and microphone to capture the user's facial expressions and voice in real time. The input consists of camera video and audio data. Facial expressions are analyzed from the camera video using OpenCV's facial recognition technology, and the audio data is analyzed using the Google Speech-to-Text API. The output consists of analyzed facial expression data and audio-to-text data.

[0741] Step 2:

[0742] The device analyzes facial expression data and voice-to-text data and sends it to the server. The data is sent to the cloud server via a secure communication protocol. The input is facial expression data and voice-to-text data, and this data is sent directly to the server as output.

[0743] Step 3:

[0744] The server analyzes the facial expression data and voice / text data it receives and uses a machine learning algorithm to evaluate the user's emotional state. This analysis determines the degree of stress and motivation the user is experiencing. The input is facial expression data and voice / text data, and the output is evaluation data indicating the emotional state.

[0745] Step 4:

[0746] The server generates appropriate support activities based on the user's emotional state. It determines the actions necessary for the user's stress relief and efficient task completion. The input is emotional state evaluation data, and the output is instruction data indicating the content of the support activity.

[0747] Step 5:

[0748] The server generates instruction data for support activities and sends it to the home robot, which then carries it out. This is done through seamless communication over the network, with the robot interacting with the user through voice and actions. The input is the instruction data sent from the server, and the output is the specific support actions performed by the robot within the home.

[0749] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0750] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0751] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0752] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0753] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0754] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0755] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0756] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0757] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0758] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0759] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0760] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0761] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0762] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0763] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0764] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0765] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0766] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0767] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0768] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0769] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0770] The following is further disclosed regarding the embodiments described above.

[0771] (Claim 1)

[0772] Means for obtaining user usage trends,

[0773] A means for identifying repetitive administrative tasks based on the aforementioned usage trends,

[0774] A means for generating instructions to automate the aforementioned repetitive administrative tasks,

[0775] A means of providing the aforementioned instructions to the user and performing automation,

[0776] A means of monitoring the progress of an organization's work and detecting anomalies,

[0777] A system that includes this.

[0778] (Claim 2)

[0779] The system according to claim 1, comprising means for analyzing past work data and generating work improvement suggestions.

[0780] (Claim 3)

[0781] The system according to claim 1, comprising means for analyzing movement patterns in real time and proposing an efficient movement path.

[0782] "Example 1"

[0783] (Claim 1)

[0784] Means for monitoring user actions and collecting information,

[0785] A means of using a computational model for analyzing the collected information,

[0786] A means for identifying recurring business activities based on the aforementioned analysis results,

[0787] A means for generating instructions to automate the aforementioned repetitive business actions,

[0788] A means of providing the aforementioned instructions to the user and performing automation,

[0789] A means of monitoring the progress of an organization's operations and detecting anomalies,

[0790] A means of sending warnings based on anomaly detection and suggesting work improvements,

[0791] A system that includes this.

[0792] (Claim 2)

[0793] The system according to claim 1, comprising means for analyzing past business data and proposing business improvements.

[0794] (Claim 3)

[0795] The system according to claim 1, comprising means for analyzing movement behavior in real time and proposing an efficient route.

[0796] "Application Example 1"

[0797] (Claim 1)

[0798] Means for monitoring and recording user actions,

[0799] A means for visualizing the progress of the work based on the aforementioned operation,

[0800] Means for detecting abnormalities in the progress and issuing warnings,

[0801] Means for generating optimization instructions for machine motion and displaying them on a visual device,

[0802] Means for executing the optimization instruction,

[0803] A system that includes this.

[0804] (Claim 2)

[0805] The system according to claim 1, comprising means for collecting motion data of work equipment and generating motion patterns for efficiency improvement.

[0806] (Claim 3)

[0807] The system according to claim 1, comprising means for analyzing past equipment movements and visually presenting suggestions for business improvement.

[0808] "Example 2 of combining an emotion engine"

[0809] (Claim 1)

[0810] A means of monitoring the user's operation history and obtaining usage trends,

[0811] A means for identifying recurring work tasks based on the aforementioned usage patterns and emotional states,

[0812] A means for generating instructions to automate the aforementioned repetitive business tasks,

[0813] A means of providing the aforementioned instructions to the user and performing automation,

[0814] A means of analyzing a user's emotional state in real time using emotion recognition technology,

[0815] A means for adjusting the priority of tasks based on the aforementioned analysis results and notifying the need for breaks,

[0816] A means of monitoring the progress of an organization's operations and detecting anomalies,

[0817] A system that includes this.

[0818] (Claim 2)

[0819] The system according to claim 1, comprising means for analyzing past operation data and sentiment data to generate work improvement suggestions.

[0820] (Claim 3)

[0821] The system according to claim 1, comprising means for analyzing movement patterns in real time and proposing efficient movement strategies.

[0822] "Application example 2 when combining with an emotional engine"

[0823] (Claim 1)

[0824] Means for obtaining user usage trends,

[0825] A means for identifying repetitive administrative tasks based on the aforementioned usage trends,

[0826] A means for generating instructions to automate the aforementioned repetitive administrative tasks,

[0827] A means of providing the aforementioned instructions to the user and performing automation,

[0828] A means of monitoring the progress of an organization's work and detecting anomalies,

[0829] A means of recognizing the emotional state of users in real time and proposing support activities based on that,

[0830] A system that includes this.

[0831] (Claim 2)

[0832] The system according to claim 1, comprising means for analyzing past work data and sentiment data to generate work improvement suggestions.

[0833] (Claim 3)

[0834] The system according to claim 1, comprising means for analyzing emotional states within the household and proposing efficient daily activities. [Explanation of symbols]

[0835] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means for obtaining user usage trends, A means for identifying repetitive administrative tasks based on the aforementioned usage trends, A means for generating instructions to automate the aforementioned repetitive administrative tasks, A means of providing the aforementioned instructions to the user and performing automation, A means of monitoring the progress of an organization's work and detecting anomalies, A system that includes this.

2. The system according to claim 1, comprising means for analyzing past work data and generating work improvement suggestions.

3. The system according to claim 1, comprising means for analyzing movement patterns in real time and proposing an efficient movement path.

Images