system

Abstract

We provide the system. [Solution] Means for collecting a series of operations on the user's computer device, A means for analyzing collected operational data and identifying repetitive operations, A means for generating automation suggestions for identified repetitive tasks, A means for performing learning using the aforementioned operational data and constructing a generative model specialized for the business, A means of presenting automation proposals to users and executing approved automations, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In enterprises, the personalization of business and the decline in business efficiency have become problems. In particular, a lot of time is spent on simple tasks that are routinely repeated, and human errors are also likely to occur. As a result, the efficiency of the entire business is reduced, which causes a decline in productivity. There is a need to solve this problem, improve business efficiency, and prevent personalization.

Means for Solving the Problems

[0005] This invention provides a system that identifies repetitive operations by collecting and analyzing the operation data of a computer device. This system generates and presents automation suggestions for the identified repetitive tasks. It also utilizes operation data to build task-specific generative models, thereby improving work efficiency. Automated tasks are approved and executed by the user, preventing reliance on individual expertise and improving work efficiency. Furthermore, the generative models are updated based on user feedback, enabling continuous improvement. In addition, operation data is encrypted during communication to ensure security.

[0006] A "computer device" is a device equipped with hardware and software that possesses computing power and is capable of performing instructed calculations and processes.

[0007] "Operation data" refers to recorded information about a series of operations performed by a user on a computer device.

[0008] A "repetitive task" is a task in which the same or similar operations are repeated regularly.

[0009] An "automation suggestion" is a proposed work procedure or tool that the system automatically provides to improve and streamline identified repetitive tasks.

[0010] A "generative model" is a machine learning model specialized for a particular task, which uses knowledge learned from data to provide suggestions and support for that task.

[0011] "Security" refers to the means and methods used to ensure the confidentiality, integrity, and availability of data and systems, and to prevent unauthorized access and data destruction.

[0012] Encryption is a technology that transforms data into a form that cannot be easily understood by third parties, and is a means of protecting confidential information.

[0013] "Feedback" refers to user opinions and reactions regarding the system's performance and output results, and is information used to improve the system. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] In an embodiment of this invention, a client program for acquiring operation data is first executed on the terminal. The terminal monitors user actions such as application startup, window switching, file operations, and input content in the background and records them as logs. This log data is periodically sent to the server.

[0036] The server analyzes the received operation data. This data analysis uses machine learning algorithms and pattern recognition techniques to identify operations that are repeated under specific conditions, i.e., repetitive tasks. The server also has a function to automatically generate automation suggestions to improve the efficiency of these identified repetitive tasks.

[0037] The generative model is also trained on the server side. The server effectively learns from operational data and builds a generative model specific to the business. This generative model is used to support optimal suggestions and automation execution according to specific business tasks.

[0038] Users review automation suggestions presented by the server on their screens. These suggestions can be reviewed, modified, and approved by the user. Approved automation tasks are executed on the terminal, thereby improving work efficiency.

[0039] For example, if a user sends emails to multiple customers every week, the server can detect this repetitive operation, automatically create email templates, and generate suggestions to automate the email sending process based on a schedule. By approving these suggestions, the user can save time and improve work efficiency.

[0040] The overall flow of this system is designed with security in mind, and operation data is transmitted encrypted, making it superior from an information protection standpoint. Furthermore, it is possible to update the generative model based on user feedback and improve the system's accuracy.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The terminal launches a client program on the user's computer. This program runs in the background, monitoring and logging all user actions in real time, such as application usage, file access, window switching, and data entry.

[0044] Step 2:

[0045] The terminal aggregates the collected operation logs at regular intervals and sends the encrypted data to the server using a secure communication protocol.

[0046] Step 3:

[0047] The server stores the received operation data in a database for analysis. The data is primarily analyzed using machine learning algorithms to identify specific patterns and repetitive tasks using frequent pattern mining techniques.

[0048] Step 4:

[0049] The server can generate appropriate automation suggestions based on identified repetitive tasks. These suggestions can be created, for example, in the form of macros or scripts, and presented as specific actions to improve the user's work efficiency.

[0050] Step 5:

[0051] Users can review automated proposals sent from the server on their own devices. They can then review the proposals and, if necessary, approve them or request modifications.

[0052] Step 6:

[0053] For approved proposals, the terminal will automatically begin execution according to the specified schedule. This execution process includes user confirmation and error handling.

[0054] Step 7:

[0055] Users evaluate the results of the automated processes and provide feedback. This feedback is sent to the server, which uses it to refine the generative model and improve the accuracy of future suggestions.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] In business operations using information processing equipment, the need for repetitive tasks and increased efficiency is growing. However, traditional methods require users to manually implement efficiency improvements, which is time-consuming and labor-intensive, hindering overall business efficiency. There is a strong desire to improve this situation by automating repetitive tasks and enhancing work efficiency.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes means for processing acquired operation information and identifying repetitive operations, means for creating work efficiency suggestions for the identified repetitive tasks, and means for constructing business-specific generation characteristics. This enables the automation of repetitive tasks for users and efficient business operations.

[0061] An "information processing device" is an electronic device used to collect, analyze, and process data.

[0062] "Operation information" refers to data related to a series of actions and operations performed by a user on an information processing device.

[0063] A "server" is a computing system connected to a network that processes and provides data in response to requests from other devices.

[0064] "Repeated operations" refer to actions in which a user performs the same or similar operation multiple times on an information processing device.

[0065] "Work efficiency improvement proposals" refer to suggestions or methods for performing identified repetitive tasks more efficiently.

[0066] "Generative characteristics" are functions and models built based on data obtained from operational information to support the optimization of tasks in specific business processes.

[0067] "Encryption" is a technology that protects data from unauthorized access by transforming it in a predetermined way, making it in a format that cannot be easily understood by third parties.

[0068] "Information protection" refers to the efforts and measures taken to protect important data, such as personal information and confidential information, from unauthorized use and leakage.

[0069] The invention is configured as follows in an embodiment for carrying it out.

[0070] The terminal runs a client program to acquire a series of actions and operation information performed by the user. This terminal is a computing device running on an operating system, such as a personal computer or a notebook computer. This program monitors application launches, window navigation, file editing, and text input in the background and records the acquired operation information as a log. The log is periodically sent to the server without the user's knowledge. The operation information is encrypted during this transmission to protect the information.

[0071] The server analyzes the received operation information. This analysis uses machine learning algorithms and pattern recognition techniques to identify a series of repetitive operations, or repetitive tasks. For the identified repetitive tasks, the server generates suggestions for improving work efficiency. This generation process is based on available generative AI models and utilizes the information to build business-specific generation characteristics.

[0072] Users can view work efficiency suggestions presented by the server on a display device. These suggestions can be reviewed by the user, modified as needed, or approved. Approved efficiency tasks are executed on the terminal, thereby achieving work efficiency improvements.

[0073] As a concrete example, consider a user's weekly task of sending emails to multiple customers. The server detects this repetitive operation, automatically creates email templates, and generates suggestions to automate the scheduled email sending process. By approving these suggestions, the user can save time on creation and sending, improving work efficiency.

[0074] An example of a prompt is, "Please suggest ways to automate sending weekly recurring emails." Using this, users can receive automation suggestions quickly and effectively.

[0075] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0076] Step 1:

[0077] The terminal acquires user activity information. Specifically, the terminal monitors application launches, window switching, file operations, and text input in the background. The data obtained through this monitoring (activity information) becomes the input. The output is log data recording this information. This log data is temporarily stored within the terminal.

[0078] Step 2:

[0079] The terminal encrypts the collected log data and periodically sends it to the server. Specifically, the terminal performs encryption at regular time intervals and sends the data through secure communication with the server. The input to this step is the log data generated in the previous step, and the output is the encrypted log data sent to the server.

[0080] Step 3:

[0081] The server decrypts and analyzes the received encrypted operation data. The server uses machine learning algorithms and pattern recognition techniques for this analysis. Specifically, after decrypting the data, the server performs an analysis to identify specific repetitive patterns. The input for this step is encrypted log data, and the output is an identified list of repetitive tasks.

[0082] Step 4:

[0083] The server generates suggestions for improving work efficiency based on the repetitive tasks identified through analysis. Specifically, it uses a generative AI model to automatically create suggestions for efficiency improvements. The input for this step is the list of repetitive tasks created in the previous step, and the output is the suggestions for improving work efficiency presented to the user.

[0084] Step 5:

[0085] Users review the efficiency improvement suggestions presented by the server and approve them as needed. Users then review the suggestions and determine if they are suitable for their specific tasks. The input is the efficiency improvement suggestions from the server, and the output is the approved or revised suggestions.

[0086] Step 6:

[0087] The terminal executes automated tasks approved by the user. Specifically, the terminal performs automated processes according to a schedule based on the user's agreement. The input is the approved work efficiency suggestion, and the output is the completion of the automated task.

[0088] Step 7:

[0089] The server updates the generative AI model based on the automated tasks performed and user feedback. Specifically, it analyzes the collected feedback and retrains the model to improve its accuracy. The input for this step is the feedback and execution data, and the output is the updated generative AI model.

[0090] (Application Example 1)

[0091] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0092] In work environments where repetitive operations are frequently performed, it is difficult for users to find the optimal way to efficiently automate tasks. Furthermore, in the operation of machinery, environments where the same procedures are repeated each time can lead to decreased work efficiency and increased human error. To prevent these decreases in work efficiency and reduce the burden on users, the introduction and rapid implementation of effective automation proposals are required. This necessitates improving productivity in industrial environments handling large-scale and complex tasks.

[0093] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0094] In this invention, the server includes means for collecting a series of operations on the user's computer; means for identifying repetitive tasks of a work machine and proposing automation based on the collected operation information; and means for notifying the user of the proposal to their personal information terminal. This makes it possible to easily identify repetitive operations in the work environment and achieve efficient automation.

[0095] A "computer" is a device that processes information and performs various operations based on user instructions.

[0096] "Operational information" refers to data relating to the inputs, outputs, and all associated activities when a user uses a computing device.

[0097] "Repetitive work" refers to tasks in which the same operations are repeated at regular intervals.

[0098] An "automation proposal" is a presentation of methods or procedures for implementing automation, generated based on collected operational information.

[0099] "Working machinery" refers to equipment and robots used in industrial environments and manufacturing sites to perform specific tasks.

[0100] A "portable information terminal" is a device that allows a user to process, display, and send / receive information in a portable form, and mainly refers to smartphones and tablets.

[0101] To implement this invention, a system consisting mainly of a server, a user's personal information terminal, and a work machine is used. The system includes a client program that collects a series of operations performed by the user on the user's computer and transmits them to the server.

[0102] The server executes machine learning algorithms using Python to analyze user data. This analysis identifies repetitive tasks and patterns for generating automation suggestions. In particular, frameworks such as TENSORFLOW® and PyTorch are used for data analysis to build generative models. The generated automation suggestions are notified to the user's mobile device.

[0103] The user's mobile device, developed using Swift (iOS) or Kotlin (Android®), displays automation suggestions from the server on its interface. The user reviews these suggestions and makes modifications or approvals as needed. Approved suggestions are sent back to the server, and the automation of the work machine is executed. The work machine is controlled by a program stored in ROM and operates based on the improved procedures.

[0104] As a concrete example, suppose a factory assembles the same parts every day. The system detects this repetitive work, automatically generates an optimized assembly procedure manual, and presents it to the manager as a suggestion. If the manager approves the suggestion, the machinery can automatically perform the assembly work based on the suggested procedure.

[0105] An example of a prompt is, "Identify repetitive patterns from the operation logs of a factory robot and propose effective methods for automating them." Using this prompt, the generative AI model will provide more accurate automation suggestions.

[0106] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0107] Step 1:

[0108] The terminal records a series of operations performed by the user on the computing device. Specifically, it monitors and collects operation information such as application launches, window switching, and input content. In this process, user operation data is obtained as input and recorded in the log as operation information.

[0109] Step 2:

[0110] The terminal periodically sends the collected operation information to the server. In this process, encrypted operation information is sent as output from the terminal and then input to the server.

[0111] Step 3:

[0112] The server analyzes the received operation information. This analysis uses machine learning algorithms (e.g., TensorFlow) to identify patterns in the data and pinpoint repetitive tasks. The input is operation information sent from the terminal, and the output is a list of identified repetitive tasks. During this process, the data is cleaned and normalized to transform it into a format suitable for the model.

[0113] Step 4:

[0114] The server proposes automation of tasks based on identified repetitive tasks. Using a generative AI model, it generates automation suggestions that allow users to improve efficiency. A list of identified repetitive tasks is used as input, and automation suggestions are generated as output.

[0115] Step 5:

[0116] The server sends the generated automation proposal to the user's mobile device. At this stage, the automation proposal is generated as output from the server and input as a notification to the mobile device.

[0117] Step 6:

[0118] Users review automation proposals on their mobile devices. Specifically, they check the proposals, make revisions, and approve them. The user's actions serve as input, and the approved automation tasks are determined as output.

[0119] Step 7:

[0120] The server sends approved automation tasks to the work machine for execution. The automation tasks are generated as output from the server, and the work machine receives them as input and begins operation. At this time, the work machine performs the work efficiently according to a pre-installed control program.

[0121] Through these steps, the system can reduce the burden on users and improve work efficiency.

[0122] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0123] In an embodiment of this invention, a client program is first executed on the terminal to monitor user operations and collect logs. The terminal uses an emotion engine to estimate the user's emotional state in real time, along with the user's interface usage. This emotion engine calculates emotional indicators such as stress and satisfaction using, for example, the user's input speed, mouse click patterns, and eye-tracking information on the screen.

[0124] The collected operation logs and emotional data are periodically encrypted and sent to the server. The server analyzes this data to identify repetitive tasks performed by the user and generates automation suggestions tailored to the user's emotional state. These suggestions are appropriately adjusted to the user's emotional state; for example, if the user's stress level is high, automation to reduce the burden of operations will be proactively implemented.

[0125] Users can review, approve, or modify suggestions from the server on their own devices. Approved suggestions are executed on the device according to a schedule, and the execution is adjusted as needed based on feedback from the sentiment engine.

[0126] For example, if a user is working during a busy period, the emotion engine can recognize the user's stress and revise automated suggestions to simplify work processes or suggest breaks using timers. In this way, flexible suggestions tailored to emotions are possible, reducing the user's workload.

[0127] Furthermore, based on user feedback and sentiment data, the server can train its generative model, continuously improving the accuracy of future suggestions. This system enhances user work efficiency, prevents reliance on individual expertise in tasks, and supports sentiment management.

[0128] The following describes the processing flow.

[0129] Step 1:

[0130] The terminal launches client programs in the background and collects user activity data. This includes the applications used, key press events, mouse movements and clicks, and screen scrolling.

[0131] Step 2:

[0132] The device runs an emotion engine to estimate the user's emotional state. The emotion engine analyzes information such as operation data, eye tracking, and input speed to evaluate the user's stress level and satisfaction level in real time.

[0133] Step 3:

[0134] The device encrypts the collected operational and emotional data and sends it to the server using a secure communication protocol. This process is performed periodically, and the data is sent in batches.

[0135] Step 4:

[0136] The server analyzes the received data and detects repetitive tasks under specific conditions. Frequent pattern mining and machine learning algorithms are used for data analysis.

[0137] Step 5:

[0138] The server generates automation suggestions based on the analysis results and the user's emotional state. If stress levels are high, it suggests ways to reduce the burden; if satisfaction levels are low, it suggests ways to improve work efficiency.

[0139] Step 6:

[0140] Users review the automation suggestions presented on their device. They can then review the suggestions and give instructions for approval or modification.

[0141] Step 7:

[0142] The device executes automated suggestions approved by the user according to a schedule. During execution, it monitors the user's emotional state and adjusts the execution as needed.

[0143] Step 8:

[0144] Users evaluate the results of automated suggestions and send feedback to the server. This feedback is used to train the generative model and improve the accuracy of future suggestions.

[0145] (Example 2)

[0146] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0147] In the use of modern computing devices, inefficient operation and a lack of adequate consideration of emotional states are factors that hinder users from working effectively and increase their stress levels. Furthermore, the failure to implement appropriate automation tailored to individual users often leads to decreased work efficiency.

[0148] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0149] In this invention, the server includes means for monitoring the user's actions on the computing device and collecting a series of activity logs, means for analyzing the recorded activity logs and recognizing repetitive operations, and means for proposing automation for the recognized repetitive tasks. This enables efficient automation suggestions and work adjustments that take into account the user's individual emotional state.

[0150] "User" refers to an individual or organization that operates a computing device and generates and uses data.

[0151] A "computing device" refers to an electronic device used to process digital data and improve the efficiency of business and personal activities.

[0152] An "activity log" refers to a dataset that records the history of user operations on a computing device.

[0153] "Emotional data" refers to information obtained from input speed, mouse operation, eye movements, etc., to determine the user's psychological state.

[0154] "Repetitive operations" refer to specific work procedures that a user frequently repeats on a computing device.

[0155] An "automation suggestion" refers to an optimized task processing plan presented to reduce the user's operational burden and improve work efficiency.

[0156] A "generative model" refers to an algorithmic model that learns from user behavior logs and sentiment data, and has the function of generating suggestions for automating tasks.

[0157] "Encryption" refers to a technology that algorithmically transforms data to enhance security and prevent unauthorized access.

[0158] To implement this invention, a client program for monitoring user operations is first executed on the terminal. This program records a series of operations performed by the user on the terminal in real time and collects them as log data. The terminal requires sensors and interfaces to acquire information such as the user's input speed, mouse click patterns, and eye-tracking information, and is equipped with an emotion engine that estimates the user's emotional state using the collected data.

[0159] The device quantifies the user's emotional state and calculates it as an indicator of stress, satisfaction, and other factors. Furthermore, by using an emotion engine, it's possible to identify situations in which users experience stress and gain insights to improve the user experience.

[0160] The collected operation logs and sentiment data are encrypted using AES (Advanced Encryption Standard) and periodically sent to the server. The server receives this data and performs behavioral analysis. Based on the received data, it identifies repetitive operations that the user frequently performs and generates automation suggestions for those repetitive tasks. These suggestions are then refined using a generative AI model to propose the optimal automation method based on the user's emotional state.

[0161] The user receives automation suggestions from the server on their device. These suggestions are reviewed by the user, approved, or modified, and approved suggestions are executed according to a schedule. For example, if a user is working during a busy period, the emotion engine can assess the user's stress level and recommend efficient work execution or appropriate breaks through automation suggestions. Such a prompt might read, "If the user is experiencing high stress during work, please generate automation suggestions to improve work efficiency."

[0162] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0163] Step 1:

[0164] The terminal launches a client program that monitors user activity. Specific inputs include the user's keyboard input speed, mouse click patterns, and on-screen eye-tracking information. This data is recorded as an activity log and used to obtain information about the user's interface usage. The output is a collected activity log.

[0165] Step 2:

[0166] The device activates its built-in emotion engine. It uses the activity log obtained in step 1 as input. By analyzing this activity log, it estimates the user's emotional state, specifically stress and satisfaction levels. Data processing involves pattern recognition of the input and feeding it into an emotion model, resulting in the output of emotional data.

[0167] Step 3:

[0168] The device encrypts the collected operation logs and sentiment data. The output data from Step 1 and Step 2 are used as input data. Encryption algorithms such as AES (Advanced Encryption Standard) are applied to obtain encrypted data as output. This data is stored in a secure format.

[0169] Step 4:

[0170] The terminal transfers encrypted data to the server. Transfers occur at regular time intervals or in response to increases in data volume. It receives encrypted data as input and transmits it over the network using a communication protocol. The output is data packets received by the server.

[0171] Step 5:

[0172] The server decrypts the received encrypted data and begins analysis. The encrypted data obtained in step 4 is used as input. After decryption, the server analyzes behavioral patterns to identify repetitive operations frequently performed by the user. Data calculations involve pattern matching and algorithmic analysis to output the identified repetitive operations.

[0173] Step 6:

[0174] The server uses a generative AI model to generate automation suggestions for identified repetitive operations. It uses the operation data and sentiment data obtained in step 5 as input. It performs calculations based on the generative AI model and outputs appropriate automation suggestions based on the sentiment.

[0175] Step 7:

[0176] The user reviews the automation proposal sent from the server on their terminal. They receive the automation proposal data as input. The user approves or modifies this proposal to match their specific business workflow. The finalized proposal is returned as output.

[0177] Step 8:

[0178] The terminal executes the user-approved proposal. It uses the automation proposal finalized in step 7 as input. It automates the workflow based on the schedule, incorporating feedback from the sentiment engine as needed. The output includes the executed automation tasks and their results.

[0179] (Application Example 2)

[0180] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0181] In the current work environment, workers' emotional states are not monitored in real time, making it difficult to improve work efficiency based on their emotions. Furthermore, repetitive tasks performed by workers are not identified, resulting in insufficient suggestions for automation and improvement. This leads to increased workload for workers and decreased productivity.

[0182] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0183] In this invention, the server includes means for aggregating a series of operations on the user's computing device, means for analyzing the aggregated operation information and recognizing repetitive operations, and means for detecting the worker's operating speed and voice and evaluating their emotional state. This makes it possible to propose optimization of the work environment and automation of tasks based on the worker's emotions.

[0184] "User computing device" refers to an electronic device used by a user to perform tasks and manage data, and is used as a user interface.

[0185] "Operation information" refers to data on a series of operations performed by a user via a computing device, and is collected in the form of logs.

[0186] "Repetitive operations" are actions or tasks that users perform repeatedly according to a specific pattern, and are activities that can be automated.

[0187] A "predictive model" is an analytical model built based on collected operational information and used to improve the efficiency of business processes and propose improvements.

[0188] "Means of creation" refers to the process of providing functions and methods for processing information and generating proposals.

[0189] "Emotional state" refers to the psychological state and degree of physical stress experienced by workers, and is a factor that affects work efficiency and workload.

[0190] "Adjustments or suggestions for the work environment" refer to improvement measures or suggestions for changes provided to workers in order to optimize their working environment, with the aim of improving work efficiency.

[0191] A description of embodiments for carrying out this invention will be given.

[0192] The system aggregates operational information generated by the worker's computing device and uses this information to identify repetitive operations. This allows for the suggestion of automation aimed at improving work efficiency. Furthermore, the device can sense the worker's movement speed and voice, and evaluate their emotional state in real time.

[0193] The main hardware of this system is an industrial robot equipped with sensor functions, such as a camera sensor or microphone. For software, "OpenCV" is used for image processing, and AI models such as "TensorFlow" are used for data analysis. The collected data is transferred to a cloud server via "AWS® Lambda," and the models are trained using "AWS SageMaker."

[0194] Based on the estimated emotional state, the server suggests optimized work environment adjustments to reduce the worker's burden. These suggestions aim to maximize work efficiency and may include automating operations or suggesting break times. For example, if the server determines that a worker is excessively fatigued, the robot may automatically adjust its work speed.

[0195] An example of a prompt from a generated AI model is: "Based on worker behavior and voice data, please tell me how to determine working conditions and stress levels and suggest the optimal work procedures and break times." This prompt is used to deepen understanding of how the system makes decisions.

[0196] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0197] Step 1:

[0198] The terminal collects user activity information in real time. This information includes keyboard input, mouse movements, and click events. The timestamp and frequency of each activity event are also recorded and stored in a database.

[0199] Step 2:

[0200] The terminal recognizes repetitive operations from the aggregated operation information. Here, statistical analysis and machine learning techniques are used to analyze the data in order to detect specific operation patterns. As a result, the identified repetitive operations are output as a list, which serves as input for the next step.

[0201] Step 3:

[0202] The server generates automation suggestions based on a list of repetitive operations. A generative AI model is used to identify parts of the operation that can be automated. Recommended automation steps are then generated and sent to the user's terminal.

[0203] Step 4:

[0204] The terminal detects the worker's movement speed and voice, and evaluates their emotional state. Sensors collect movement speed and voice tone, and an AI model determines stress levels and concentration levels. The obtained emotional data is sent to a server.

[0205] Step 5:

[0206] The server adjusts or suggests changes to the work environment based on data evaluating the user's emotional state. Specifically, this might include increasing automation when stress levels are high or recommending breaks. The generated suggestions are sent to the terminal and presented for the user to review.

[0207] Step 6:

[0208] The user reviews the automation proposal and approves or modifies it as needed. The user's input determines the final automation steps, which are then executed on the terminal. The final results are sent to the server as feedback.

[0209] Step 7:

[0210] The server continuously learns and updates its generative AI model based on collected feedback and sentiment data. This makes it possible to improve the accuracy of future suggestions.

[0211] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0212] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0213] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0214] [Second Embodiment]

[0215] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0216] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0217] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0218] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0219] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0220] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0221] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0222] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0223] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0224] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0225] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0226] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0227] In an embodiment of this invention, a client program for acquiring operation data is first executed on the terminal. The terminal monitors user actions such as application startup, window switching, file operations, and input content in the background and records them as logs. This log data is periodically sent to the server.

[0228] The server analyzes the received operation data. This data analysis uses machine learning algorithms and pattern recognition techniques to identify operations that are repeated under specific conditions, i.e., repetitive tasks. The server also has a function to automatically generate automation suggestions to improve the efficiency of these identified repetitive tasks.

[0229] The generative model is also trained on the server side. The server effectively learns from operational data and builds a generative model specific to the business. This generative model is used to support optimal suggestions and automation execution according to specific business tasks.

[0230] Users review automation suggestions presented by the server on their screens. These suggestions can be reviewed, modified, and approved by the user. Approved automation tasks are executed on the terminal, thereby improving work efficiency.

[0231] For example, if a user sends emails to multiple customers every week, the server can detect this repetitive operation, automatically create email templates, and generate suggestions to automate the email sending process based on a schedule. By approving these suggestions, the user can save time and improve work efficiency.

[0232] The overall flow of this system is designed with security in mind, and operation data is transmitted encrypted, making it superior from an information protection standpoint. Furthermore, it is possible to update the generative model based on user feedback and improve the system's accuracy.

[0233] The following describes the processing flow.

[0234] Step 1:

[0235] The terminal launches a client program on the user's computer. This program runs in the background, monitoring and logging all user actions in real time, such as application usage, file access, window switching, and data entry.

[0236] Step 2:

[0237] The terminal aggregates the collected operation logs at regular intervals and sends the encrypted data to the server using a secure communication protocol.

[0238] Step 3:

[0239] The server stores the received operation data in a database for analysis. The data is primarily analyzed using machine learning algorithms to identify specific patterns and repetitive tasks using frequent pattern mining techniques.

[0240] Step 4:

[0241] The server can generate appropriate automation suggestions based on identified repetitive tasks. These suggestions can be created, for example, in the form of macros or scripts, and presented as specific actions to improve the user's work efficiency.

[0242] Step 5:

[0243] Users can review automated proposals sent from the server on their own devices. They can then review the proposals and, if necessary, approve them or request modifications.

[0244] Step 6:

[0245] For approved proposals, the terminal will automatically begin execution according to the specified schedule. This execution process includes user confirmation and error handling.

[0246] Step 7:

[0247] Users evaluate the results of the automated processes and provide feedback. This feedback is sent to the server, which uses it to refine the generative model and improve the accuracy of future suggestions.

[0248] (Example 1)

[0249] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0250] In business operations using information processing equipment, the need for repetitive tasks and increased efficiency is growing. However, traditional methods require users to manually implement efficiency improvements, which is time-consuming and labor-intensive, hindering overall business efficiency. There is a strong desire to improve this situation by automating repetitive tasks and enhancing work efficiency.

[0251] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0252] In this invention, the server includes means for processing acquired operation information and identifying repetitive operations, means for creating work efficiency suggestions for the identified repetitive tasks, and means for constructing business-specific generation characteristics. This enables the automation of repetitive tasks for users and efficient business operations.

[0253] An "information processing device" is an electronic device used to collect, analyze, and process data.

[0254] "Operation information" refers to data related to a series of actions and operations performed by a user on an information processing device.

[0255] A "server" is a computing system connected to a network that processes and provides data in response to requests from other devices.

[0256] "Repeated operations" refer to actions in which a user performs the same or similar operation multiple times on an information processing device.

[0257] "Work efficiency improvement proposals" refer to suggestions or methods for performing identified repetitive tasks more efficiently.

[0258] "Generative characteristics" are functions and models built based on data obtained from operational information to support the optimization of tasks in specific business processes.

[0259] "Encryption" is a technology that protects data from unauthorized access by transforming it in a predetermined way, making it in a format that cannot be easily understood by third parties.

[0260] "Information protection" refers to the efforts and measures taken to protect important data, such as personal information and confidential information, from unauthorized use and leakage.

[0261] The invention is configured as follows in an embodiment for carrying it out.

[0262] The terminal runs a client program to acquire a series of actions and operation information performed by the user. This terminal is a computing device running on an operating system, such as a personal computer or a notebook computer. This program monitors application launches, window navigation, file editing, and text input in the background and records the acquired operation information as a log. The log is periodically sent to the server without the user's knowledge. The operation information is encrypted during this transmission to protect the information.

[0263] The server analyzes the received operation information. This analysis uses machine learning algorithms and pattern recognition techniques to identify a series of repetitive operations, or repetitive tasks. For the identified repetitive tasks, the server generates suggestions for improving work efficiency. This generation process is based on available generative AI models and utilizes the information to build business-specific generation characteristics.

[0264] Users can view work efficiency suggestions presented by the server on a display device. These suggestions can be reviewed by the user, modified as needed, or approved. Approved efficiency tasks are executed on the terminal, thereby achieving work efficiency improvements.

[0265] As a concrete example, consider a user's weekly task of sending emails to multiple customers. The server detects this repetitive operation, automatically creates email templates, and generates suggestions to automate the scheduled email sending process. By approving these suggestions, the user can save time on creation and sending, improving work efficiency.

[0266] An example of a prompt is, "Please suggest ways to automate sending weekly recurring emails." Using this, users can receive automation suggestions quickly and effectively.

[0267] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0268] Step 1:

[0269] The terminal acquires user activity information. Specifically, the terminal monitors application launches, window switching, file operations, and text input in the background. The data obtained through this monitoring (activity information) becomes the input. The output is log data recording this information. This log data is temporarily stored within the terminal.

[0270] Step 2:

[0271] The terminal encrypts the collected log data and periodically sends it to the server. Specifically, the terminal performs encryption at regular time intervals and sends the data through secure communication with the server. The input to this step is the log data generated in the previous step, and the output is the encrypted log data sent to the server.

[0272] Step 3:

[0273] The server decrypts and analyzes the received encrypted operation data. The server uses machine learning algorithms and pattern recognition techniques for this analysis. Specifically, after decrypting the data, the server performs an analysis to identify specific repetitive patterns. The input for this step is encrypted log data, and the output is an identified list of repetitive tasks.

[0274] Step 4:

[0275] The server generates suggestions for improving work efficiency based on the repetitive tasks identified through analysis. Specifically, it uses a generative AI model to automatically create suggestions for efficiency improvements. The input for this step is the list of repetitive tasks created in the previous step, and the output is the suggestions for improving work efficiency presented to the user.

[0276] Step 5:

[0277] Users review the efficiency improvement suggestions presented by the server and approve them as needed. Users then review the suggestions and determine if they are suitable for their specific tasks. The input is the efficiency improvement suggestions from the server, and the output is the approved or revised suggestions.

[0278] Step 6:

[0279] The terminal executes automated tasks approved by the user. As a specific operation, the terminal performs an automated process according to a schedule based on the user's consent. The input is an approved work efficiency improvement proposal, and the output is the completion of the automated work.

[0280] Step 7:

[0281] The server updates the generated AI model based on the executed automated tasks and feedback from the user. As a specific operation, it analyzes the collected feedback and performs relearning to improve the model's accuracy. The input for this step is feedback and execution data, and the output is the updated generated AI model.

[0282] (Application Example 1)

[0283] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0284] In a work environment where repetitive operations are frequently performed, it is difficult for users to find the optimal method to automate work efficiently. Also, in the operation of work machinery, in an environment where the same procedure is repeated every time, work efficiency decreases and there is a possibility of causing human errors. Furthermore, in order to prevent these decreases in work efficiency and reduce the burden on the user, the introduction of effective automation proposals and their rapid execution are required. Thereby, it is necessary to improve productivity in an industrial environment that handles large-scale and complex work.

[0285] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0286] In this invention, the server includes means for collecting a series of operations on the user's computing device, means for identifying repetitive operations of the working machine based on the collected operation information and proposing automation, and means for notifying the user's portable information terminal of the said proposal. Thereby, it becomes possible to easily identify repetitive operations in the working environment and realize efficient automation.

[0287] A "computing device" is a device that processes information and performs various operations based on user instructions.

[0288] "Operation information" is data related to inputs, outputs, and all associated activities when the user uses the computing device.

[0289] "Repetitive operation" refers to an operation in which operations with the same content are repeated at regular intervals.

[0290] "Automation proposal" is the presentation of a method or procedure for implementing automation, generated based on the collected operation information.

[0291] A "working machine" refers to equipment or robots used in an industrial environment or manufacturing site to perform specific tasks.

[0292] A "portable information terminal" is a device that can process, display, and transmit and receive information in a form that can be carried by the user, mainly referring to smartphones and tablets.

[0293] To implement this invention, a system mainly composed of a server, the user's portable information terminal, and the working machine is used. The system includes a client program that collects a series of operations performed by the user on the user's computing device and transmits them to the server.

[0294] The server executes machine learning algorithms using Python to analyze operational information. This analysis identifies repetitive tasks and patterns for generating automation suggestions. In particular, frameworks such as TensorFlow and PyTorch are used for data analysis to build generative models. The generated automation suggestions are notified to the user's mobile device.

[0295] The user's mobile device, developed using Swift (iOS) or Kotlin (Android), displays automation suggestions from the server on its interface. The user reviews these suggestions and makes modifications or approvals as needed. Approved suggestions are sent back to the server, and the automation of the work machine is executed. The work machine is controlled by a program stored in ROM and operates based on the improved procedures.

[0296] As a concrete example, suppose a factory assembles the same parts every day. The system detects this repetitive work, automatically generates an optimized assembly procedure manual, and presents it to the manager as a suggestion. If the manager approves the suggestion, the machinery can automatically perform the assembly work based on the suggested procedure.

[0297] An example of a prompt is, "Identify repetitive patterns from the operation logs of a factory robot and propose effective methods for automating them." Using this prompt, the generative AI model will provide more accurate automation suggestions.

[0298] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0299] Step 1:

[0300] The terminal records a series of operations performed by the user on the computing device. Specifically, it monitors and collects operation information such as application launches, window switching, and input content. In this process, user operation data is obtained as input and recorded in the log as operation information.

[0301] Step 2:

[0302] The terminal periodically sends the collected operation information to the server. Here, the encrypted operation information is sent as the output from the terminal and takes the form of being input into the server.

[0303] Step 3:

[0304] The server analyzes the received operation information. In this analysis, a machine learning algorithm (e.g., TensorFlow) is used to identify patterns in the data and specify repetitive tasks. The input is the operation information sent from the terminal, and the output is a list of the specified repetitive tasks. In this process, data cleaning and normalization are performed to convert it into a form suitable for the model.

[0305] Step 4:

[0306] The server proposes automation of the work based on the specified repetitive tasks. A generative AI model is used to generate automation proposals that enable the user to improve efficiency. The list of the specified repetitive tasks is used as the input, and automation proposals are generated as the output.

[0307] Step 5:

[0308] The server sends the generated automation proposals to the user's mobile information terminal. At this stage, the automation proposals are generated as the output from the server and are input as notifications to the mobile information terminal.

[0309] Step 6:

[0310] The user checks the automation proposals on the mobile information terminal. In particular, the user checks the proposed content and makes corrections or approvals. The user's operation serves as the input, and the approved automation tasks are determined as the output.

[0311] Step 7:

[0312] The server sends approved automation tasks to the work machine for execution. The automation tasks are generated as output from the server, and the work machine receives them as input and begins operation. At this time, the work machine performs the work efficiently according to a pre-installed control program.

[0313] Through these steps, the system can reduce the burden on users and improve work efficiency.

[0314] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0315] In an embodiment of this invention, a client program is first executed on the terminal to monitor user operations and collect logs. The terminal uses an emotion engine to estimate the user's emotional state in real time, along with the user's interface usage. This emotion engine calculates emotional indicators such as stress and satisfaction using, for example, the user's input speed, mouse click patterns, and eye-tracking information on the screen.

[0316] The collected operation logs and emotional data are periodically encrypted and sent to the server. The server analyzes this data to identify repetitive tasks performed by the user and generates automation suggestions tailored to the user's emotional state. These suggestions are appropriately adjusted to the user's emotional state; for example, if the user's stress level is high, automation to reduce the burden of operations will be proactively implemented.

[0317] Users can review, approve, or modify suggestions from the server on their own devices. Approved suggestions are executed on the device according to a schedule, and the execution is adjusted as needed based on feedback from the sentiment engine.

[0318] For example, if a user is working during a busy period, the emotion engine can recognize the user's stress and revise automated suggestions to simplify work processes or suggest breaks using timers. In this way, flexible suggestions tailored to emotions are possible, reducing the user's workload.

[0319] Furthermore, based on user feedback and sentiment data, the server can train its generative model, continuously improving the accuracy of future suggestions. This system enhances user work efficiency, prevents reliance on individual expertise in tasks, and supports sentiment management.

[0320] The following describes the processing flow.

[0321] Step 1:

[0322] The terminal launches client programs in the background and collects user activity data. This includes the applications used, key press events, mouse movements and clicks, and screen scrolling.

[0323] Step 2:

[0324] The device runs an emotion engine to estimate the user's emotional state. The emotion engine analyzes information such as operation data, eye tracking, and input speed to evaluate the user's stress level and satisfaction level in real time.

[0325] Step 3:

[0326] The device encrypts the collected operational and emotional data and sends it to the server using a secure communication protocol. This process is performed periodically, and the data is sent in batches.

[0327] Step 4:

[0328] The server analyzes the received data and detects repetitive tasks under specific conditions. Frequent pattern mining and machine learning algorithms are used for data analysis.

[0329] Step 5:

[0330] The server generates automation suggestions based on the analysis results and the user's emotional state. If stress levels are high, it suggests ways to reduce the burden; if satisfaction levels are low, it suggests ways to improve work efficiency.

[0331] Step 6:

[0332] Users review the automation suggestions presented on their device. They can then review the suggestions and give instructions for approval or modification.

[0333] Step 7:

[0334] The device executes automated suggestions approved by the user according to a schedule. During execution, it monitors the user's emotional state and adjusts the execution as needed.

[0335] Step 8:

[0336] Users evaluate the results of automated suggestions and send feedback to the server. This feedback is used to train the generative model and improve the accuracy of future suggestions.

[0337] (Example 2)

[0338] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0339] In the use of modern computing devices, inefficient operation and a lack of adequate consideration of emotional states are factors that hinder users from working effectively and increase their stress levels. Furthermore, the failure to implement appropriate automation tailored to individual users often leads to decreased work efficiency.

[0340] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0341] In this invention, the server includes means for monitoring the user's actions on the computing device and collecting a series of activity logs, means for analyzing the recorded activity logs and recognizing repetitive operations, and means for proposing automation for the recognized repetitive tasks. This enables efficient automation suggestions and work adjustments that take into account the user's individual emotional state.

[0342] "User" refers to an individual or organization that operates a computing device and generates and uses data.

[0343] A "computing device" refers to an electronic device used to process digital data and improve the efficiency of business and personal activities.

[0344] An "activity log" refers to a dataset that records the history of user operations on a computing device.

[0345] "Emotional data" refers to information obtained from input speed, mouse operation, eye movements, etc., to determine the user's psychological state.

[0346] "Repetitive operations" refer to specific work procedures that a user frequently repeats on a computing device.

[0347] An "automation suggestion" refers to an optimized task processing plan presented to reduce the user's operational burden and improve work efficiency.

[0348] A "generative model" refers to an algorithmic model that learns from user behavior logs and sentiment data, and has the function of generating suggestions for automating tasks.

[0349] "Encryption" refers to a technology that algorithmically transforms data to enhance security and prevent unauthorized access.

[0350] To implement this invention, a client program for monitoring user operations is first executed on the terminal. This program records a series of operations performed by the user on the terminal in real time and collects them as log data. The terminal requires sensors and interfaces to acquire information such as the user's input speed, mouse click patterns, and eye-tracking information, and is equipped with an emotion engine that estimates the user's emotional state using the collected data.

[0351] The device quantifies the user's emotional state and calculates it as an indicator of stress, satisfaction, and other factors. Furthermore, by using an emotion engine, it's possible to identify situations in which users experience stress and gain insights to improve the user experience.

[0352] The collected operation logs and sentiment data are encrypted using AES (Advanced Encryption Standard) and periodically sent to the server. The server receives this data and performs behavioral analysis. Based on the received data, it identifies repetitive operations that the user frequently performs and generates automation suggestions for those repetitive tasks. These suggestions are then refined using a generative AI model to propose the optimal automation method based on the user's emotional state.

[0353] The user receives automation suggestions from the server on their device. These suggestions are reviewed by the user, approved, or modified, and approved suggestions are executed according to a schedule. For example, if a user is working during a busy period, the emotion engine can assess the user's stress level and recommend efficient work execution or appropriate breaks through automation suggestions. Such a prompt might read, "If the user is experiencing high stress during work, please generate automation suggestions to improve work efficiency."

[0354] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0355] Step 1:

[0356] The terminal launches a client program that monitors user activity. Specific inputs include the user's keyboard input speed, mouse click patterns, and on-screen eye-tracking information. This data is recorded as an activity log and used to obtain information about the user's interface usage. The output is a collected activity log.

[0357] Step 2:

[0358] The device activates its built-in emotion engine. It uses the activity log obtained in step 1 as input. By analyzing this activity log, it estimates the user's emotional state, specifically stress and satisfaction levels. Data processing involves pattern recognition of the input and feeding it into an emotion model, resulting in the output of emotional data.

[0359] Step 3:

[0360] The device encrypts the collected operation logs and sentiment data. The output data from Step 1 and Step 2 are used as input data. Encryption algorithms such as AES (Advanced Encryption Standard) are applied to obtain encrypted data as output. This data is stored in a secure format.

[0361] Step 4:

[0362] The terminal transfers encrypted data to the server. Transfers occur at regular time intervals or in response to increases in data volume. It receives encrypted data as input and transmits it over the network using a communication protocol. The output is data packets received by the server.

[0363] Step 5:

[0364] The server decrypts the received encrypted data and begins analysis. The encrypted data obtained in step 4 is used as input. After decryption, the server analyzes behavioral patterns to identify repetitive operations frequently performed by the user. Data calculations involve pattern matching and algorithmic analysis to output the identified repetitive operations.

[0365] Step 6:

[0366] The server uses a generative AI model to generate automation suggestions for identified repetitive operations. It uses the operation data and sentiment data obtained in step 5 as input. It performs calculations based on the generative AI model and outputs appropriate automation suggestions based on the sentiment.

[0367] Step 7:

[0368] The user reviews the automation proposal sent from the server on their terminal. They receive the automation proposal data as input. The user approves or modifies this proposal to match their specific business workflow. The finalized proposal is returned as output.

[0369] Step 8:

[0370] The terminal executes the user-approved proposal. It uses the automation proposal finalized in step 7 as input. It automates the workflow based on the schedule, incorporating feedback from the sentiment engine as needed. The output includes the executed automation tasks and their results.

[0371] (Application Example 2)

[0372] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0373] In the current work environment, workers' emotional states are not monitored in real time, making it difficult to improve work efficiency based on their emotions. Furthermore, repetitive tasks performed by workers are not identified, resulting in insufficient suggestions for automation and improvement. This leads to increased workload for workers and decreased productivity.

[0374] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0375] In this invention, the server includes means for aggregating a series of operations on the user's computing device, means for analyzing the aggregated operation information and recognizing repetitive operations, and means for detecting the worker's operating speed and voice and evaluating their emotional state. This makes it possible to propose optimization of the work environment and automation of tasks based on the worker's emotions.

[0376] "User computing device" refers to an electronic device used by a user to perform tasks and manage data, and is used as a user interface.

[0377] "Operation information" refers to data on a series of operations performed by a user via a computing device, and is collected in the form of logs.

[0378] "Repetitive operations" are actions or tasks that users perform repeatedly according to a specific pattern, and are activities that can be automated.

[0379] A "predictive model" is an analytical model built based on collected operational information and used to improve the efficiency of business processes and propose improvements.

[0380] "Means of creation" refers to the process of providing functions and methods for processing information and generating proposals.

[0381] "Emotional state" refers to the psychological state and degree of physical stress experienced by workers, and is a factor that affects work efficiency and workload.

[0382] "Adjustments or suggestions for the work environment" refer to improvement measures or suggestions for changes provided to workers in order to optimize their working environment, with the aim of improving work efficiency.

[0383] A description of embodiments for carrying out this invention will be given.

[0384] The system aggregates operational information generated by the worker's computing device and uses this information to identify repetitive operations. This allows for the suggestion of automation aimed at improving work efficiency. Furthermore, the device can sense the worker's movement speed and voice, and evaluate their emotional state in real time.

[0385] The main hardware of this system is an industrial robot equipped with sensor functions, such as a camera sensor or microphone. For software, "OpenCV" is used for image processing, and AI models such as "TensorFlow" are used for data analysis. The collected data is transferred to a cloud server via "AWS Lambda," and the models are trained using "AWS SageMaker."

[0386] Based on the estimated emotional state, the server suggests optimized work environment adjustments to reduce the worker's burden. These suggestions aim to maximize work efficiency and may include automating operations or suggesting break times. For example, if the server determines that a worker is excessively fatigued, the robot may automatically adjust its work speed.

[0387] An example of a prompt from a generated AI model is: "Based on worker behavior and voice data, please tell me how to determine working conditions and stress levels and suggest the optimal work procedures and break times." This prompt is used to deepen understanding of how the system makes decisions.

[0388] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0389] Step 1:

[0390] The terminal collects user activity information in real time. This information includes keyboard input, mouse movements, and click events. The timestamp and frequency of each activity event are also recorded and stored in a database.

[0391] Step 2:

[0392] The terminal recognizes repetitive operations from the aggregated operation information. Here, statistical analysis and machine learning techniques are used to analyze the data in order to detect specific operation patterns. As a result, the identified repetitive operations are output as a list, which serves as input for the next step.

[0393] Step 3:

[0394] The server generates automation suggestions based on a list of repetitive operations. A generative AI model is used to identify parts of the operation that can be automated. Recommended automation steps are then generated and sent to the user's terminal.

[0395] Step 4:

[0396] The terminal detects the worker's movement speed and voice, and evaluates their emotional state. Sensors collect movement speed and voice tone, and an AI model determines stress levels and concentration levels. The obtained emotional data is sent to a server.

[0397] Step 5:

[0398] The server adjusts or suggests changes to the work environment based on data evaluating the user's emotional state. Specifically, this might include increasing automation when stress levels are high or recommending breaks. The generated suggestions are sent to the terminal and presented for the user to review.

[0399] Step 6:

[0400] The user reviews the automation proposal and approves or modifies it as needed. The user's input determines the final automation steps, which are then executed on the terminal. The final results are sent to the server as feedback.

[0401] Step 7:

[0402] The server continuously learns and updates its generative AI model based on collected feedback and sentiment data. This makes it possible to improve the accuracy of future suggestions.

[0403] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0404] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0405] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0406] [Third Embodiment]

[0407] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0408] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0409] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0410] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0411] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0412] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0413] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0414] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0415] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0416] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0417] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0418] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0419] In an embodiment of this invention, a client program for acquiring operation data is first executed on the terminal. The terminal monitors user actions such as application startup, window switching, file operations, and input content in the background and records them as logs. This log data is periodically sent to the server.

[0420] The server analyzes the received operation data. This data analysis uses machine learning algorithms and pattern recognition techniques to identify operations that are repeated under specific conditions, i.e., repetitive tasks. The server also has a function to automatically generate automation suggestions to improve the efficiency of these identified repetitive tasks.

[0421] The generative model is also trained on the server side. The server effectively learns from operational data and builds a generative model specific to the business. This generative model is used to support optimal suggestions and automation execution according to specific business tasks.

[0422] Users review automation suggestions presented by the server on their screens. These suggestions can be reviewed, modified, and approved by the user. Approved automation tasks are executed on the terminal, thereby improving work efficiency.

[0423] For example, if a user sends emails to multiple customers every week, the server can detect this repetitive operation, automatically create email templates, and generate suggestions to automate the email sending process based on a schedule. By approving these suggestions, the user can save time and improve work efficiency.

[0424] The overall flow of this system is designed with security in mind, and operation data is transmitted encrypted, making it superior from an information protection standpoint. Furthermore, it is possible to update the generative model based on user feedback and improve the system's accuracy.

[0425] The following describes the processing flow.

[0426] Step 1:

[0427] The terminal launches a client program on the user's computer. This program runs in the background, monitoring and logging all user actions in real time, such as application usage, file access, window switching, and data entry.

[0428] Step 2:

[0429] The terminal aggregates the collected operation logs at regular intervals and sends the encrypted data to the server using a secure communication protocol.

[0430] Step 3:

[0431] The server stores the received operation data in a database for analysis. The data is primarily analyzed using machine learning algorithms to identify specific patterns and repetitive tasks using frequent pattern mining techniques.

[0432] Step 4:

[0433] The server can generate appropriate automation suggestions based on identified repetitive tasks. These suggestions can be created, for example, in the form of macros or scripts, and presented as specific actions to improve the user's work efficiency.

[0434] Step 5:

[0435] Users can review automated proposals sent from the server on their own devices. They can then review the proposals and, if necessary, approve them or request modifications.

[0436] Step 6:

[0437] For approved proposals, the terminal will automatically begin execution according to the specified schedule. This execution process includes user confirmation and error handling.

[0438] Step 7:

[0439] Users evaluate the results of the automated processes and provide feedback. This feedback is sent to the server, which uses it to refine the generative model and improve the accuracy of future suggestions.

[0440] (Example 1)

[0441] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0442] In business operations using information processing equipment, the need for repetitive tasks and increased efficiency is growing. However, traditional methods require users to manually implement efficiency improvements, which is time-consuming and labor-intensive, hindering overall business efficiency. There is a strong desire to improve this situation by automating repetitive tasks and enhancing work efficiency.

[0443] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0444] In this invention, the server includes means for processing acquired operation information and identifying repetitive operations, means for creating work efficiency suggestions for the identified repetitive tasks, and means for constructing business-specific generation characteristics. This enables the automation of repetitive tasks for users and efficient business operations.

[0445] An "information processing device" is an electronic device used to collect, analyze, and process data.

[0446] "Operation information" refers to data related to a series of actions and operations performed by a user on an information processing device.

[0447] A "server" is a computing system connected to a network that processes and provides data in response to requests from other devices.

[0448] "Repeated operations" refer to actions in which a user performs the same or similar operation multiple times on an information processing device.

[0449] "Work efficiency improvement proposals" refer to suggestions or methods for performing identified repetitive tasks more efficiently.

[0450] "Generative characteristics" are functions and models built based on data obtained from operational information to support the optimization of tasks in specific business processes.

[0451] "Encryption" is a technology that protects data from unauthorized access by transforming it in a predetermined way, making it in a format that cannot be easily understood by third parties.

[0452] "Information protection" refers to the efforts and measures taken to protect important data, such as personal information and confidential information, from unauthorized use and leakage.

[0453] The invention is configured as follows in an embodiment for carrying it out.

[0454] The terminal runs a client program to acquire a series of actions and operation information performed by the user. This terminal is a computing device running on an operating system, such as a personal computer or a notebook computer. This program monitors application launches, window navigation, file editing, and text input in the background and records the acquired operation information as a log. The log is periodically sent to the server without the user's knowledge. The operation information is encrypted during this transmission to protect the information.

[0455] The server analyzes the received operation information. This analysis uses machine learning algorithms and pattern recognition techniques to identify a series of repetitive operations, or repetitive tasks. For the identified repetitive tasks, the server generates suggestions for improving work efficiency. This generation process is based on available generative AI models and utilizes the information to build business-specific generation characteristics.

[0456] Users can view work efficiency suggestions presented by the server on a display device. These suggestions can be reviewed by the user, modified as needed, or approved. Approved efficiency tasks are executed on the terminal, thereby achieving work efficiency improvements.

[0457] As a concrete example, consider a user's weekly task of sending emails to multiple customers. The server detects this repetitive operation, automatically creates email templates, and generates suggestions to automate the scheduled email sending process. By approving these suggestions, the user can save time on creation and sending, improving work efficiency.

[0458] An example of a prompt is, "Please suggest ways to automate sending weekly recurring emails." Using this, users can receive automation suggestions quickly and effectively.

[0459] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0460] Step 1:

[0461] The terminal acquires user activity information. Specifically, the terminal monitors application launches, window switching, file operations, and text input in the background. The data obtained through this monitoring (activity information) becomes the input. The output is log data recording this information. This log data is temporarily stored within the terminal.

[0462] Step 2:

[0463] The terminal encrypts the collected log data and periodically sends it to the server. Specifically, the terminal performs encryption at regular time intervals and sends the data through secure communication with the server. The input to this step is the log data generated in the previous step, and the output is the encrypted log data sent to the server.

[0464] Step 3:

[0465] The server decrypts and analyzes the received encrypted operation data. The server uses machine learning algorithms and pattern recognition techniques for this analysis. Specifically, after decrypting the data, the server performs an analysis to identify specific repetitive patterns. The input for this step is encrypted log data, and the output is an identified list of repetitive tasks.

[0466] Step 4:

[0467] The server generates suggestions for improving work efficiency based on the repetitive tasks identified through analysis. Specifically, it uses a generative AI model to automatically create suggestions for efficiency improvements. The input for this step is the list of repetitive tasks created in the previous step, and the output is the suggestions for improving work efficiency presented to the user.

[0468] Step 5:

[0469] Users review the efficiency improvement suggestions presented by the server and approve them as needed. Users then review the suggestions and determine if they are suitable for their specific tasks. The input is the efficiency improvement suggestions from the server, and the output is the approved or revised suggestions.

[0470] Step 6:

[0471] The terminal executes automated tasks approved by the user. Specifically, the terminal performs automated processes according to a schedule based on the user's agreement. The input is the approved work efficiency suggestion, and the output is the completion of the automated task.

[0472] Step 7:

[0473] The server updates the generative AI model based on the automated tasks performed and user feedback. Specifically, it analyzes the collected feedback and retrains the model to improve its accuracy. The input for this step is the feedback and execution data, and the output is the updated generative AI model.

[0474] (Application Example 1)

[0475] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0476] In work environments where repetitive operations are frequently performed, it is difficult for users to find the optimal way to efficiently automate tasks. Furthermore, in the operation of machinery, environments where the same procedures are repeated each time can lead to decreased work efficiency and increased human error. To prevent these decreases in work efficiency and reduce the burden on users, the introduction and rapid implementation of effective automation proposals are required. This necessitates improving productivity in industrial environments handling large-scale and complex tasks.

[0477] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0478] In this invention, the server includes means for collecting a series of operations on the user's computer; means for identifying repetitive tasks of a work machine and proposing automation based on the collected operation information; and means for notifying the user of the proposal to their personal information terminal. This makes it possible to easily identify repetitive operations in the work environment and achieve efficient automation.

[0479] A "computer" is a device that processes information and performs various operations based on user instructions.

[0480] "Operational information" refers to data relating to the inputs, outputs, and all associated activities when a user uses a computing device.

[0481] "Repetitive work" refers to tasks in which the same operations are repeated at regular intervals.

[0482] An "automation proposal" is a presentation of methods or procedures for implementing automation, generated based on collected operational information.

[0483] "Working machinery" refers to equipment and robots used in industrial environments and manufacturing sites to perform specific tasks.

[0484] A "portable information terminal" is a device that allows a user to process, display, and send / receive information in a portable form, and mainly refers to smartphones and tablets.

[0485] To implement this invention, a system consisting mainly of a server, a user's personal information terminal, and a work machine is used. The system includes a client program that collects a series of operations performed by the user on the user's computer and transmits them to the server.

[0486] The server executes machine learning algorithms using Python to analyze operational information. This analysis identifies repetitive tasks and patterns for generating automation suggestions. In particular, frameworks such as TensorFlow and PyTorch are used for data analysis to build generative models. The generated automation suggestions are notified to the user's mobile device.

[0487] The user's mobile device, developed using Swift (iOS) or Kotlin (Android), displays automation suggestions from the server on its interface. The user reviews these suggestions and makes modifications or approvals as needed. Approved suggestions are sent back to the server, and the automation of the work machine is executed. The work machine is controlled by a program stored in ROM and operates based on the improved procedures.

[0488] As a concrete example, suppose a factory assembles the same parts every day. The system detects this repetitive work, automatically generates an optimized assembly procedure manual, and presents it to the manager as a suggestion. If the manager approves the suggestion, the machinery can automatically perform the assembly work based on the suggested procedure.

[0489] An example of a prompt is, "Identify repetitive patterns from the operation logs of a factory robot and propose effective methods for automating them." Using this prompt, the generative AI model will provide more accurate automation suggestions.

[0490] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0491] Step 1:

[0492] The terminal records a series of operations performed by the user on the computing device. Specifically, it monitors and collects operation information such as application launches, window switching, and input content. In this process, user operation data is obtained as input and recorded in the log as operation information.

[0493] Step 2:

[0494] The terminal periodically sends the collected operation information to the server. Here, encrypted operation information is sent as output from the terminal and then input to the server.

[0495] Step 3:

[0496] The server analyzes the received operation information. This analysis uses machine learning algorithms (e.g., TensorFlow) to identify patterns in the data and pinpoint repetitive tasks. The input is operation information sent from the terminal, and the output is a list of identified repetitive tasks. During this process, the data is cleaned and normalized to transform it into a format suitable for the model.

[0497] Step 4:

[0498] The server proposes automation of tasks based on identified repetitive tasks. Using a generative AI model, it generates automation suggestions that allow users to improve efficiency. A list of identified repetitive tasks is used as input, and automation suggestions are generated as output.

[0499] Step 5:

[0500] The server sends the generated automation proposal to the user's mobile device. At this stage, the automation proposal is generated as output from the server and input as a notification to the mobile device.

[0501] Step 6:

[0502] Users review automation proposals on their mobile devices. Specifically, they check the proposals, make revisions, and approve them. The user's actions serve as input, and the approved automation tasks are determined as output.

[0503] Step 7:

[0504] The server sends approved automation tasks to the work machine for execution. The automation tasks are generated as output from the server, and the work machine receives them as input and begins operation. At this time, the work machine performs the work efficiently according to a pre-installed control program.

[0505] Through these steps, the system can reduce the burden on users and improve work efficiency.

[0506] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0507] In an embodiment of this invention, a client program is first executed on the terminal to monitor user operations and collect logs. The terminal uses an emotion engine to estimate the user's emotional state in real time, along with the user's interface usage. This emotion engine calculates emotional indicators such as stress and satisfaction using, for example, the user's input speed, mouse click patterns, and eye-tracking information on the screen.

[0508] The collected operation logs and emotional data are periodically encrypted and sent to the server. The server analyzes this data to identify repetitive tasks performed by the user and generates automation suggestions tailored to the user's emotional state. These suggestions are appropriately adjusted to the user's emotional state; for example, if the user's stress level is high, automation to reduce the burden of operations will be proactively implemented.

[0509] Users can review, approve, or modify suggestions from the server on their own devices. Approved suggestions are executed on the device according to a schedule, and the execution is adjusted as needed based on feedback from the sentiment engine.

[0510] For example, if a user is working during a busy period, the emotion engine can recognize the user's stress and revise automated suggestions to simplify work processes or suggest breaks using timers. In this way, flexible suggestions tailored to emotions are possible, reducing the user's workload.

[0511] Furthermore, based on user feedback and sentiment data, the server can train its generative model, continuously improving the accuracy of future suggestions. This system enhances user work efficiency, prevents reliance on individual expertise in tasks, and supports sentiment management.

[0512] The following describes the processing flow.

[0513] Step 1:

[0514] The terminal launches client programs in the background and collects user activity data. This includes the applications used, key press events, mouse movements and clicks, and screen scrolling.

[0515] Step 2:

[0516] The device runs an emotion engine to estimate the user's emotional state. The emotion engine analyzes information such as operation data, eye tracking, and input speed to evaluate the user's stress level and satisfaction level in real time.

[0517] Step 3:

[0518] The device encrypts the collected operational and emotional data and sends it to the server using a secure communication protocol. This process is performed periodically, and the data is sent in batches.

[0519] Step 4:

[0520] The server analyzes the received data and detects repetitive tasks under specific conditions. Frequent pattern mining and machine learning algorithms are used for data analysis.

[0521] Step 5:

[0522] The server generates automation suggestions based on the analysis results and the user's emotional state. If stress levels are high, it suggests ways to reduce the burden; if satisfaction levels are low, it suggests ways to improve work efficiency.

[0523] Step 6:

[0524] Users review the automation suggestions presented on their device. They can then review the suggestions and give instructions for approval or modification.

[0525] Step 7:

[0526] The device executes automated suggestions approved by the user according to a schedule. During execution, it monitors the user's emotional state and adjusts the execution as needed.

[0527] Step 8:

[0528] Users evaluate the results of automated suggestions and send feedback to the server. This feedback is used to train the generative model and improve the accuracy of future suggestions.

[0529] (Example 2)

[0530] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0531] In the use of modern computing devices, inefficient operation and a lack of adequate consideration of emotional states are factors that hinder users from working effectively and increase their stress levels. Furthermore, the failure to implement appropriate automation tailored to individual users often leads to decreased work efficiency.

[0532] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0533] In this invention, the server includes means for monitoring the user's actions on the computing device and collecting a series of activity logs, means for analyzing the recorded activity logs and recognizing repetitive operations, and means for proposing automation for the recognized repetitive tasks. This enables efficient automation suggestions and work adjustments that take into account the user's individual emotional state.

[0534] "User" refers to an individual or organization that operates a computing device and generates and uses data.

[0535] A "computing device" refers to an electronic device used to process digital data and improve the efficiency of business and personal activities.

[0536] An "activity log" refers to a dataset that records the history of user operations on a computing device.

[0537] "Emotional data" refers to information obtained from input speed, mouse operation, eye movements, etc., to determine the user's psychological state.

[0538] "Repetitive operations" refer to specific work procedures that a user frequently repeats on a computing device.

[0539] An "automation suggestion" refers to an optimized task processing plan presented to reduce the user's operational burden and improve work efficiency.

[0540] A "generative model" refers to an algorithmic model that learns from user behavior logs and sentiment data, and has the function of generating suggestions for automating tasks.

[0541] "Encryption" refers to a technology that algorithmically transforms data to enhance security and prevent unauthorized access.

[0542] To implement this invention, a client program for monitoring user operations is first executed on the terminal. This program records a series of operations performed by the user on the terminal in real time and collects them as log data. The terminal requires sensors and interfaces to acquire information such as the user's input speed, mouse click patterns, and eye-tracking information, and is equipped with an emotion engine that estimates the user's emotional state using the collected data.

[0543] The device quantifies the user's emotional state and calculates it as an indicator of stress, satisfaction, and other factors. Furthermore, by using an emotion engine, it's possible to identify situations in which users experience stress and gain insights to improve the user experience.

[0544] The collected operation logs and sentiment data are encrypted using AES (Advanced Encryption Standard) and periodically sent to the server. The server receives this data and performs behavioral analysis. Based on the received data, it identifies repetitive operations that the user frequently performs and generates automation suggestions for those repetitive tasks. These suggestions are then refined using a generative AI model to propose the optimal automation method based on the user's emotional state.

[0545] The user receives automation suggestions from the server on their device. These suggestions are reviewed by the user, approved, or modified, and approved suggestions are executed according to a schedule. For example, if a user is working during a busy period, the emotion engine can assess the user's stress level and recommend efficient work execution or appropriate breaks through automation suggestions. Such a prompt might read, "If the user is experiencing high stress during work, please generate automation suggestions to improve work efficiency."

[0546] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0547] Step 1:

[0548] The terminal launches a client program that monitors user activity. Specific inputs include the user's keyboard input speed, mouse click patterns, and on-screen eye-tracking information. This data is recorded as an activity log and used to obtain information about the user's interface usage. The output is a collected activity log.

[0549] Step 2:

[0550] The device activates its built-in emotion engine. It uses the activity log obtained in step 1 as input. By analyzing this activity log, it estimates the user's emotional state, specifically stress and satisfaction levels. Data processing involves pattern recognition of the input and feeding it into an emotion model, resulting in the output of emotional data.

[0551] Step 3:

[0552] The device encrypts the collected operation logs and sentiment data. The output data from Step 1 and Step 2 are used as input data. Encryption algorithms such as AES (Advanced Encryption Standard) are applied to obtain encrypted data as output. This data is stored in a secure format.

[0553] Step 4:

[0554] The terminal transfers encrypted data to the server. Transfers occur at regular time intervals or in response to increases in data volume. It receives encrypted data as input and transmits it over the network using a communication protocol. The output is data packets received by the server.

[0555] Step 5:

[0556] The server decrypts the received encrypted data and begins analysis. The encrypted data obtained in step 4 is used as input. After decryption, the server analyzes behavioral patterns to identify repetitive operations frequently performed by the user. Data calculations involve pattern matching and algorithmic analysis to output the identified repetitive operations.

[0557] Step 6:

[0558] The server uses a generative AI model to generate automation suggestions for identified repetitive operations. It uses the operation data and sentiment data obtained in step 5 as input. It performs calculations based on the generative AI model and outputs appropriate automation suggestions based on the sentiment.

[0559] Step 7:

[0560] The user reviews the automation proposal sent from the server on their terminal. They receive the automation proposal data as input. The user approves or modifies this proposal to match their specific business workflow. The finalized proposal is returned as output.

[0561] Step 8:

[0562] The terminal executes the user-approved proposal. It uses the automation proposal finalized in step 7 as input. It automates the workflow based on the schedule, incorporating feedback from the sentiment engine as needed. The output includes the executed automation tasks and their results.

[0563] (Application Example 2)

[0564] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0565] In the current work environment, workers' emotional states are not monitored in real time, making it difficult to improve work efficiency based on their emotions. Furthermore, repetitive tasks performed by workers are not identified, resulting in insufficient suggestions for automation and improvement. This leads to increased workload for workers and decreased productivity.

[0566] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0567] In this invention, the server includes means for aggregating a series of operations on the user's computing device, means for analyzing the aggregated operation information and recognizing repetitive operations, and means for detecting the worker's operating speed and voice and evaluating their emotional state. This makes it possible to propose optimization of the work environment and automation of tasks based on the worker's emotions.

[0568] "User computing device" refers to an electronic device used by a user to perform tasks and manage data, and is used as a user interface.

[0569] "Operation information" refers to data on a series of operations performed by a user via a computing device, and is collected in the form of logs.

[0570] "Repetitive operations" are actions or tasks that users perform repeatedly according to a specific pattern, and are activities that can be automated.

[0571] A "predictive model" is an analytical model built based on collected operational information and used to improve the efficiency of business processes and propose improvements.

[0572] "Means of creation" refers to the process of providing functions and methods for processing information and generating proposals.

[0573] "Emotional state" refers to the psychological state and degree of physical stress experienced by workers, and is a factor that affects work efficiency and workload.

[0574] "Adjustments or suggestions for the work environment" refer to improvement measures or suggestions for changes provided to workers in order to optimize their working environment, with the aim of improving work efficiency.

[0575] A description of embodiments for carrying out this invention will be given.

[0576] The system aggregates operational information generated by the worker's computing device and uses this information to identify repetitive operations. This allows for the suggestion of automation aimed at improving work efficiency. Furthermore, the device can sense the worker's movement speed and voice, and evaluate their emotional state in real time.

[0577] The main hardware of this system is an industrial robot equipped with sensor functions, such as a camera sensor or microphone. For software, "OpenCV" is used for image processing, and AI models such as "TensorFlow" are used for data analysis. The collected data is transferred to a cloud server via "AWS Lambda," and the models are trained using "AWS SageMaker."

[0578] Based on the estimated emotional state, the server suggests optimized work environment adjustments to reduce the worker's burden. These suggestions aim to maximize work efficiency and may include automating operations or suggesting break times. For example, if the server determines that a worker is excessively fatigued, the robot may automatically adjust its work speed.

[0579] An example of a prompt from a generated AI model is: "Based on worker behavior and voice data, please tell me how to determine working conditions and stress levels and suggest the optimal work procedures and break times." This prompt is used to deepen understanding of how the system makes decisions.

[0580] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0581] Step 1:

[0582] The terminal collects user activity information in real time. This information includes keyboard input, mouse movements, and click events. The timestamp and frequency of each activity event are also recorded and stored in a database.

[0583] Step 2:

[0584] The terminal recognizes repetitive operations from the aggregated operation information. Here, statistical analysis and machine learning techniques are used to analyze the data in order to detect specific operation patterns. As a result, the identified repetitive operations are output as a list, which serves as input for the next step.

[0585] Step 3:

[0586] The server generates automation suggestions based on a list of repetitive operations. A generative AI model is used to identify parts of the operation that can be automated. Recommended automation steps are then generated and sent to the user's terminal.

[0587] Step 4:

[0588] The terminal detects the worker's movement speed and voice, and evaluates their emotional state. Sensors collect movement speed and voice tone, and an AI model determines stress levels and concentration levels. The obtained emotional data is sent to a server.

[0589] Step 5:

[0590] The server adjusts or suggests changes to the work environment based on data evaluating the user's emotional state. Specifically, this might include increasing automation when stress levels are high or recommending breaks. The generated suggestions are sent to the terminal and presented for the user to review.

[0591] Step 6:

[0592] The user reviews the automation proposal and approves or modifies it as needed. The user's input determines the final automation steps, which are then executed on the terminal. The final results are sent to the server as feedback.

[0593] Step 7:

[0594] The server continuously learns and updates its generative AI model based on collected feedback and sentiment data. This makes it possible to improve the accuracy of future suggestions.

[0595] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0596] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0597] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0598] [Fourth Embodiment]

[0599] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0600] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0601] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0602] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0603] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0604] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0605] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0606] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0607] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0608] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0609] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0610] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0611] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0612] In an embodiment of this invention, a client program for acquiring operation data is first executed on the terminal. The terminal monitors user actions such as application startup, window switching, file operations, and input content in the background and records them as logs. This log data is periodically sent to the server.

[0613] The server analyzes the received operation data. This data analysis uses machine learning algorithms and pattern recognition techniques to identify operations that are repeated under specific conditions, i.e., repetitive tasks. The server also has a function to automatically generate automation suggestions to improve the efficiency of these identified repetitive tasks.

[0614] The generative model is also trained on the server side. The server effectively learns from operational data and builds a generative model specific to the business. This generative model is used to support optimal suggestions and automation execution according to specific business tasks.

[0615] Users review automation suggestions presented by the server on their screens. These suggestions can be reviewed, modified, and approved by the user. Approved automation tasks are executed on the terminal, thereby improving work efficiency.

[0616] For example, if a user sends emails to multiple customers every week, the server can detect this repetitive operation, automatically create email templates, and generate suggestions to automate the email sending process based on a schedule. By approving these suggestions, the user can save time and improve work efficiency.

[0617] The overall flow of this system is designed with security in mind, and operation data is transmitted encrypted, making it superior from an information protection standpoint. Furthermore, it is possible to update the generative model based on user feedback and improve the system's accuracy.

[0618] The following describes the processing flow.

[0619] Step 1:

[0620] The terminal launches a client program on the user's computer. This program runs in the background, monitoring and logging all user actions in real time, such as application usage, file access, window switching, and data entry.

[0621] Step 2:

[0622] The terminal aggregates the collected operation logs at regular intervals and sends the encrypted data to the server using a secure communication protocol.

[0623] Step 3:

[0624] The server stores the received operation data in a database for analysis. The data is primarily analyzed using machine learning algorithms to identify specific patterns and repetitive tasks using frequent pattern mining techniques.

[0625] Step 4:

[0626] The server can generate appropriate automation suggestions based on identified repetitive tasks. These suggestions can be created, for example, in the form of macros or scripts, and presented as specific actions to improve the user's work efficiency.

[0627] Step 5:

[0628] Users can review automated proposals sent from the server on their own devices. They can then review the proposals and, if necessary, approve them or request modifications.

[0629] Step 6:

[0630] For approved proposals, the terminal will automatically begin execution according to the specified schedule. This execution process includes user confirmation and error handling.

[0631] Step 7:

[0632] Users evaluate the results of the automated processes and provide feedback. This feedback is sent to the server, which uses it to refine the generative model and improve the accuracy of future suggestions.

[0633] (Example 1)

[0634] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0635] In business operations using information processing equipment, the need for repetitive tasks and increased efficiency is growing. However, traditional methods require users to manually implement efficiency improvements, which is time-consuming and labor-intensive, hindering overall business efficiency. There is a strong desire to improve this situation by automating repetitive tasks and enhancing work efficiency.

[0636] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0637] In this invention, the server includes means for processing acquired operation information and identifying repetitive operations, means for creating work efficiency suggestions for the identified repetitive tasks, and means for constructing business-specific generation characteristics. This enables the automation of repetitive tasks for users and efficient business operations.

[0638] An "information processing device" is an electronic device used to collect, analyze, and process data.

[0639] "Operation information" refers to data related to a series of actions and operations performed by a user on an information processing device.

[0640] A "server" is a computing system connected to a network that processes and provides data in response to requests from other devices.

[0641] "Repeated operations" refer to actions in which a user performs the same or similar operation multiple times on an information processing device.

[0642] "Work efficiency improvement proposals" refer to suggestions or methods for performing identified repetitive tasks more efficiently.

[0643] "Generative characteristics" are functions and models built based on data obtained from operational information to support the optimization of tasks in specific business processes.

[0644] "Encryption" is a technology that protects data from unauthorized access by transforming it in a predetermined way, making it in a format that cannot be easily understood by third parties.

[0645] "Information protection" refers to the efforts and measures taken to protect important data, such as personal information and confidential information, from unauthorized use and leakage.

[0646] The invention is configured as follows in an embodiment for carrying it out.

[0647] The terminal runs a client program to acquire a series of actions and operation information performed by the user. This terminal is a computing device running on an operating system, such as a personal computer or a notebook computer. This program monitors application launches, window navigation, file editing, and text input in the background and records the acquired operation information as a log. The log is periodically sent to the server without the user's knowledge. The operation information is encrypted during this transmission to protect the information.

[0648] The server analyzes the received operation information. This analysis uses machine learning algorithms and pattern recognition techniques to identify a series of repetitive operations, or repetitive tasks. For the identified repetitive tasks, the server generates suggestions for improving work efficiency. This generation process is based on available generative AI models and utilizes the information to build business-specific generation characteristics.

[0649] Users can view work efficiency suggestions presented by the server on a display device. These suggestions can be reviewed by the user, modified as needed, or approved. Approved efficiency tasks are executed on the terminal, thereby achieving work efficiency improvements.

[0650] As a concrete example, consider a user's weekly task of sending emails to multiple customers. The server detects this repetitive operation, automatically creates email templates, and generates suggestions to automate the scheduled email sending process. By approving these suggestions, the user can save time on creation and sending, improving work efficiency.

[0651] An example of a prompt is, "Please suggest ways to automate sending weekly recurring emails." Using this, users can receive automation suggestions quickly and effectively.

[0652] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0653] Step 1:

[0654] The terminal acquires user activity information. Specifically, the terminal monitors application launches, window switching, file operations, and text input in the background. The data obtained through this monitoring (activity information) becomes the input. The output is log data recording this information. This log data is temporarily stored within the terminal.

[0655] Step 2:

[0656] The terminal encrypts the collected log data and periodically sends it to the server. Specifically, the terminal performs encryption at regular time intervals and sends the data through secure communication with the server. The input to this step is the log data generated in the previous step, and the output is the encrypted log data sent to the server.

[0657] Step 3:

[0658] The server decrypts and analyzes the received encrypted operation data. The server uses machine learning algorithms and pattern recognition techniques for this analysis. Specifically, after decrypting the data, the server performs an analysis to identify specific repetitive patterns. The input for this step is encrypted log data, and the output is an identified list of repetitive tasks.

[0659] Step 4:

[0660] The server generates suggestions for improving work efficiency based on the repetitive tasks identified through analysis. Specifically, it uses a generative AI model to automatically create suggestions for efficiency improvements. The input for this step is the list of repetitive tasks created in the previous step, and the output is the suggestions for improving work efficiency presented to the user.

[0661] Step 5:

[0662] Users review the efficiency improvement suggestions presented by the server and approve them as needed. Users then review the suggestions and determine if they are suitable for their specific tasks. The input is the efficiency improvement suggestions from the server, and the output is the approved or revised suggestions.

[0663] Step 6:

[0664] The terminal executes automated tasks approved by the user. Specifically, the terminal performs automated processes according to a schedule based on the user's agreement. The input is the approved work efficiency suggestion, and the output is the completion of the automated task.

[0665] Step 7:

[0666] The server updates the generative AI model based on the automated tasks performed and user feedback. Specifically, it analyzes the collected feedback and retrains the model to improve its accuracy. The input for this step is the feedback and execution data, and the output is the updated generative AI model.

[0667] (Application Example 1)

[0668] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0669] In work environments where repetitive operations are frequently performed, it is difficult for users to find the optimal way to efficiently automate tasks. Furthermore, in the operation of machinery, environments where the same procedures are repeated each time can lead to decreased work efficiency and increased human error. To prevent these decreases in work efficiency and reduce the burden on users, the introduction and rapid implementation of effective automation proposals are required. This necessitates improving productivity in industrial environments handling large-scale and complex tasks.

[0670] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0671] In this invention, the server includes means for collecting a series of operations on the user's computer; means for identifying repetitive tasks of a work machine and proposing automation based on the collected operation information; and means for notifying the user of the proposal to their personal information terminal. This makes it possible to easily identify repetitive operations in the work environment and achieve efficient automation.

[0672] A "computer" is a device that processes information and performs various operations based on user instructions.

[0673] "Operational information" refers to data relating to the inputs, outputs, and all associated activities when a user uses a computing device.

[0674] "Repetitive work" refers to tasks in which the same operation is repeated at regular intervals.

[0675] An "automation proposal" is a presentation of methods or procedures for implementing automation, generated based on collected operational information.

[0676] "Working machinery" refers to equipment and robots used in industrial environments and manufacturing sites to perform specific tasks.

[0677] A "portable information terminal" is a device that allows a user to process, display, and send / receive information in a portable form, and mainly refers to smartphones and tablets.

[0678] To implement this invention, a system consisting mainly of a server, a user's personal information terminal, and a work machine is used. The system includes a client program that collects a series of operations performed by the user on the user's computer and transmits them to the server.

[0679] The server executes machine learning algorithms using Python to analyze operational information. This analysis identifies repetitive tasks and patterns for generating automation suggestions. In particular, frameworks such as TensorFlow and PyTorch are used for data analysis to build generative models. The generated automation suggestions are notified to the user's mobile device.

[0680] The user's mobile device, developed using Swift (iOS) or Kotlin (Android), displays automation suggestions from the server on its interface. The user reviews these suggestions and makes modifications or approvals as needed. Approved suggestions are sent back to the server, and the automation of the work machine is executed. The work machine is controlled by a program stored in ROM and operates based on the improved procedures.

[0681] As a concrete example, suppose a factory assembles the same parts every day. The system detects this repetitive work, automatically generates an optimized assembly procedure manual, and presents it to the manager as a suggestion. If the manager approves the suggestion, the machinery can automatically perform the assembly work based on the suggested procedure.

[0682] An example of a prompt is, "Identify repetitive patterns from the operation logs of a factory robot and propose effective methods for automating them." Using this prompt, the generative AI model will provide more accurate automation suggestions.

[0683] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0684] Step 1:

[0685] The terminal records a series of operations performed by the user on the computing device. Specifically, it monitors and collects operation information such as application launches, window switching, and input content. In this process, user operation data is obtained as input and recorded in the log as operation information.

[0686] Step 2:

[0687] The terminal periodically sends the collected operation information to the server. Here, encrypted operation information is sent as output from the terminal and then input to the server.

[0688] Step 3:

[0689] The server analyzes the received operation information. This analysis uses machine learning algorithms (e.g., TensorFlow) to identify patterns in the data and pinpoint repetitive tasks. The input is operation information sent from the terminal, and the output is a list of identified repetitive tasks. During this process, the data is cleaned and normalized to transform it into a format suitable for the model.

[0690] Step 4:

[0691] The server proposes automation of tasks based on identified repetitive tasks. Using a generative AI model, it generates automation suggestions that allow users to improve efficiency. A list of identified repetitive tasks is used as input, and automation suggestions are generated as output.

[0692] Step 5:

[0693] The server sends the generated automation proposal to the user's mobile device. At this stage, the automation proposal is generated as output from the server and input as a notification to the mobile device.

[0694] Step 6:

[0695] Users review automation proposals on their mobile devices. Specifically, they check the proposals, make revisions, and approve them. The user's actions serve as input, and the approved automation tasks are determined as output.

[0696] Step 7:

[0697] The server sends approved automation tasks to the work machine for execution. The automation tasks are generated as output from the server, and the work machine receives them as input and begins operation. At this time, the work machine performs the work efficiently according to a pre-installed control program.

[0698] Through these steps, the system can reduce the burden on users and improve work efficiency.

[0699] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0700] In an embodiment of this invention, a client program is first executed on the terminal to monitor user operations and collect logs. The terminal uses an emotion engine to estimate the user's emotional state in real time, along with the user's interface usage. This emotion engine calculates emotional indicators such as stress and satisfaction using, for example, the user's input speed, mouse click patterns, and eye-tracking information on the screen.

[0701] The collected operation logs and emotional data are periodically encrypted and sent to the server. The server analyzes this data to identify repetitive tasks performed by the user and generates automation suggestions tailored to the user's emotional state. These suggestions are appropriately adjusted to the user's emotional state; for example, if the user's stress level is high, automation to reduce the burden of operations will be proactively implemented.

[0702] Users can review, approve, or modify suggestions from the server on their own devices. Approved suggestions are executed on the device according to a schedule, and the execution is adjusted as needed based on feedback from the sentiment engine.

[0703] For example, if a user is working during a busy period, the emotion engine can recognize the user's stress and revise automated suggestions to simplify work processes or suggest breaks using timers. In this way, flexible suggestions tailored to emotions are possible, reducing the user's workload.

[0704] Furthermore, based on user feedback and sentiment data, the server can train its generative model, continuously improving the accuracy of future suggestions. This system enhances user work efficiency, prevents reliance on individual expertise in tasks, and supports sentiment management.

[0705] The following describes the processing flow.

[0706] Step 1:

[0707] The terminal launches client programs in the background and collects user activity data. This includes the applications used, key press events, mouse movements and clicks, and screen scrolling.

[0708] Step 2:

[0709] The device runs an emotion engine to estimate the user's emotional state. The emotion engine analyzes information such as operation data, eye tracking, and input speed to evaluate the user's stress level and satisfaction level in real time.

[0710] Step 3:

[0711] The device encrypts the collected operational and emotional data and sends it to the server using a secure communication protocol. This process is performed periodically, and the data is sent in batches.

[0712] Step 4:

[0713] The server analyzes the received data and detects repetitive tasks under specific conditions. Frequent pattern mining and machine learning algorithms are used for data analysis.

[0714] Step 5:

[0715] The server generates automation suggestions based on the analysis results and the user's emotional state. If stress levels are high, it suggests ways to reduce the burden; if satisfaction levels are low, it suggests ways to improve work efficiency.

[0716] Step 6:

[0717] Users review the automation suggestions presented on their device. They can then review the suggestions and give instructions for approval or modification.

[0718] Step 7:

[0719] The device executes automated suggestions approved by the user according to a schedule. During execution, it monitors the user's emotional state and adjusts the execution as needed.

[0720] Step 8:

[0721] Users evaluate the results of automated suggestions and send feedback to the server. This feedback is used to train the generative model and improve the accuracy of future suggestions.

[0722] (Example 2)

[0723] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0724] In the use of modern computing devices, inefficient operation and a lack of adequate consideration of emotional states are factors that hinder users from working effectively and increase their stress levels. Furthermore, the failure to implement appropriate automation tailored to individual users often leads to decreased work efficiency.

[0725] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0726] In this invention, the server includes means for monitoring the user's actions on the computing device and collecting a series of activity logs, means for analyzing the recorded activity logs and recognizing repetitive operations, and means for proposing automation for the recognized repetitive tasks. This enables efficient automation suggestions and work adjustments that take into account the user's individual emotional state.

[0727] "User" refers to an individual or organization that operates a computing device and generates and uses data.

[0728] A "computing device" refers to an electronic device used to process digital data and improve the efficiency of business and personal activities.

[0729] An "activity log" refers to a dataset that records the history of user operations on a computing device.

[0730] "Emotional data" refers to information obtained from input speed, mouse operation, eye movements, etc., to determine the user's psychological state.

[0731] "Repetitive operations" refer to specific work procedures that a user frequently repeats on a computing device.

[0732] An "automation suggestion" refers to an optimized task processing plan presented to reduce the user's operational burden and improve work efficiency.

[0733] A "generative model" refers to an algorithmic model that learns from user behavior logs and sentiment data, and has the function of generating suggestions for automating tasks.

[0734] "Encryption" refers to a technology that algorithmically transforms data to enhance security and prevent unauthorized access.

[0735] To implement this invention, a client program for monitoring user operations is first executed on the terminal. This program records a series of operations performed by the user on the terminal in real time and collects them as log data. The terminal requires sensors and interfaces to acquire information such as the user's input speed, mouse click patterns, and eye-tracking information, and is equipped with an emotion engine that estimates the user's emotional state using the collected data.

[0736] The device quantifies the user's emotional state and calculates it as an indicator of stress, satisfaction, and other factors. Furthermore, by using an emotion engine, it's possible to identify situations in which users experience stress and gain insights to improve the user experience.

[0737] The collected operation logs and sentiment data are encrypted using AES (Advanced Encryption Standard) and periodically sent to the server. The server receives this data and performs behavioral analysis. Based on the received data, it identifies repetitive operations that the user frequently performs and generates automation suggestions for those repetitive tasks. These suggestions are then refined using a generative AI model to propose the optimal automation method based on the user's emotional state.

[0738] The user receives automation suggestions from the server on their device. These suggestions are reviewed by the user, approved, or modified, and approved suggestions are executed according to a schedule. For example, if a user is working during a busy period, the emotion engine can assess the user's stress level and recommend efficient work execution or appropriate breaks through automation suggestions. Such a prompt might read, "If the user is experiencing high stress during work, please generate automation suggestions to improve work efficiency."

[0739] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0740] Step 1:

[0741] The terminal launches a client program that monitors user activity. Specific inputs include the user's keyboard input speed, mouse click patterns, and on-screen eye-tracking information. This data is recorded as an activity log and used to obtain information about the user's interface usage. The output is a collected activity log.

[0742] Step 2:

[0743] The device activates its built-in emotion engine. It uses the activity log obtained in step 1 as input. By analyzing this activity log, it estimates the user's emotional state, specifically stress and satisfaction levels. Data processing involves pattern recognition of the input and feeding it into an emotion model, resulting in the output of emotional data.

[0744] Step 3:

[0745] The device encrypts the collected operation logs and sentiment data. The output data from Step 1 and Step 2 are used as input data. Encryption algorithms such as AES (Advanced Encryption Standard) are applied to obtain encrypted data as output. This data is stored in a secure format.

[0746] Step 4:

[0747] The terminal transfers encrypted data to the server. Transfers occur at regular time intervals or in response to increases in data volume. It receives encrypted data as input and transmits it over the network using a communication protocol. The output is data packets received by the server.

[0748] Step 5:

[0749] The server decrypts the received encrypted data and begins analysis. The encrypted data obtained in step 4 is used as input. After decryption, the server analyzes behavioral patterns to identify repetitive operations frequently performed by the user. Data calculations involve pattern matching and algorithmic analysis to output the identified repetitive operations.

[0750] Step 6:

[0751] The server uses a generative AI model to generate automation suggestions for identified repetitive operations. It uses the operation data and sentiment data obtained in step 5 as input. It performs calculations based on the generative AI model and outputs appropriate automation suggestions based on the sentiment.

[0752] Step 7:

[0753] The user reviews the automation proposal sent from the server on their terminal. They receive the automation proposal data as input. The user approves or modifies this proposal to match their specific business workflow. The finalized proposal is returned as output.

[0754] Step 8:

[0755] The terminal executes the user-approved proposal. It uses the automation proposal finalized in step 7 as input. It automates the workflow based on the schedule, incorporating feedback from the sentiment engine as needed. The output includes the executed automation tasks and their results.

[0756] (Application Example 2)

[0757] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0758] In the current work environment, workers' emotional states are not monitored in real time, making it difficult to improve work efficiency based on their emotions. Furthermore, repetitive tasks performed by workers are not identified, resulting in insufficient suggestions for automation and improvement. This leads to increased workload for workers and decreased productivity.

[0759] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0760] In this invention, the server includes means for aggregating a series of operations on the user's computing device, means for analyzing the aggregated operation information and recognizing repetitive operations, and means for detecting the worker's operating speed and voice and evaluating their emotional state. This makes it possible to propose optimization of the work environment and automation of tasks based on the worker's emotions.

[0761] "User computing device" refers to an electronic device used by a user to perform tasks and manage data, and is used as a user interface.

[0762] "Operation information" refers to data on a series of operations performed by a user via a computing device, and is collected in the form of logs.

[0763] "Repetitive operations" are actions or tasks that users perform repeatedly according to a specific pattern, and are activities that can be automated.

[0764] A "predictive model" is an analytical model built based on collected operational information and used to improve the efficiency of business processes and propose improvements.

[0765] "Means of creation" refers to the process of providing functions and methods for processing information and generating proposals.

[0766] "Emotional state" refers to the psychological state and degree of physical stress experienced by workers, and is a factor that affects work efficiency and workload.

[0767] "Adjustments or suggestions for the work environment" refer to improvement measures or suggestions for changes provided to workers in order to optimize their working environment, with the aim of improving work efficiency.

[0768] A description of embodiments for carrying out this invention will be given.

[0769] The system aggregates operational information generated by the worker's computing device and uses this information to identify repetitive operations. This allows for the suggestion of automation aimed at improving work efficiency. Furthermore, the device can sense the worker's movement speed and voice, and evaluate their emotional state in real time.

[0770] The main hardware of this system is an industrial robot equipped with sensor functions, such as a camera sensor or microphone. For software, "OpenCV" is used for image processing, and AI models such as "TensorFlow" are used for data analysis. The collected data is transferred to a cloud server via "AWS Lambda," and the models are trained using "AWS SageMaker."

[0771] Based on the estimated emotional state, the server suggests optimized work environment adjustments to reduce the worker's burden. These suggestions aim to maximize work efficiency and may include automating operations or suggesting break times. For example, if the server determines that a worker is excessively fatigued, the robot may automatically adjust its work speed.

[0772] An example of a prompt from a generated AI model is: "Based on worker behavior and voice data, please tell me how to determine working conditions and stress levels and suggest the optimal work procedures and break times." This prompt is used to deepen understanding of how the system makes decisions.

[0773] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0774] Step 1:

[0775] The terminal collects user activity information in real time. This information includes keyboard input, mouse movements, and click events. The timestamp and frequency of each activity event are also recorded and stored in a database.

[0776] Step 2:

[0777] The terminal recognizes repetitive operations from the aggregated operation information. Here, statistical analysis and machine learning techniques are used to analyze the data in order to detect specific operation patterns. As a result, the identified repetitive operations are output as a list, which serves as input for the next step.

[0778] Step 3:

[0779] The server generates automation suggestions based on a list of repetitive operations. A generative AI model is used to identify parts of the operation that can be automated. Recommended automation steps are then generated and sent to the user's terminal.

[0780] Step 4:

[0781] The terminal detects the worker's movement speed and voice, and evaluates their emotional state. Sensors collect movement speed and voice tone, and an AI model determines stress levels and concentration levels. The obtained emotional data is sent to a server.

[0782] Step 5:

[0783] The server adjusts or suggests changes to the work environment based on data evaluating the user's emotional state. Specifically, this might include increasing automation when stress levels are high or recommending breaks. The generated suggestions are sent to the terminal and presented for the user to review.

[0784] Step 6:

[0785] The user reviews the automation proposal and approves or modifies it as needed. The user's input determines the final automation steps, which are then executed on the terminal. The final results are sent to the server as feedback.

[0786] Step 7:

[0787] The server continuously learns and updates its generative AI model based on collected feedback and sentiment data. This makes it possible to improve the accuracy of future suggestions.

[0788] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0789] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0790] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0791] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0792] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0793] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0794] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0795] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0796] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0797] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0798] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0799] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0800] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0801] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0802] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0803] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0804] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0805] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0806] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0807] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0808] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0809] The following is further disclosed regarding the embodiments described above.

[0810] (Claim 1)

[0811] Means for collecting a series of operations on the user's computer device,

[0812] A means for analyzing collected operational data and identifying repetitive operations,

[0813] A means for generating automation suggestions for identified repetitive tasks,

[0814] A means for performing learning using the aforementioned operational data and constructing a generative model specialized for the business,

[0815] A means of presenting automation proposals to users and executing approved automations,

[0816] A system that includes this.

[0817] (Claim 2)

[0818] The system according to claim 1, further comprising means for updating the generative model based on feedback from the user.

[0819] (Claim 3)

[0820] The system according to claim 1, further comprising means for encrypting the operation data and communicating it, with security in mind.

[0821] "Example 1"

[0822] (Claim 1)

[0823] Means for acquiring a series of operations on the user's information processing device,

[0824] A means for processing acquired operation information and identifying repeated operations,

[0825] A means for creating work efficiency suggestions for identified repetitive tasks,

[0826] A means for performing training using the aforementioned operational information and constructing generation characteristics specific to the task,

[0827] A means of presenting work efficiency improvement proposals to users and implementing the efficiency improvements once they have agreed upon,

[0828] A system that includes this.

[0829] (Claim 2)

[0830] The system according to claim 1, further comprising means for improving the generation characteristics based on evaluations from users.

[0831] (Claim 3)

[0832] The system according to claim 1, further comprising means for encrypting the operation information and communicating it, taking into consideration information protection.

[0833] "Application Example 1"

[0834] (Claim 1)

[0835] Means for collecting a series of operations on the user's computing device,

[0836] A means for analyzing collected operational information and identifying repetitive operations,

[0837] A means for generating automation suggestions for identified repetitive behaviors,

[0838] A means for performing learning using the aforementioned operational information and constructing a generative model specialized for the business,

[0839] A means of presenting automation proposals to users and executing approved automations,

[0840] A means to identify repetitive tasks of work machines based on collected operational information and propose automation,

[0841] A means of notifying the user of the aforementioned proposal via their mobile device,

[0842] A system that includes this.

[0843] (Claim 2)

[0844] The system according to claim 1, further comprising means for updating the generative model based on feedback from the user.

[0845] (Claim 3)

[0846] The system according to claim 1, further comprising means for encrypting the operation information and communicating it, with security in mind.

[0847] "Example 2 of combining an emotion engine"

[0848] (Claim 1)

[0849] A means for monitoring the user's actions on the computing device and collecting a series of activity logs,

[0850] Means for analyzing the recorded activity log and recognizing repetitive operations,

[0851] A means of proposing automation for recognized repetitive tasks,

[0852] Means for encrypting and transmitting the aforementioned behavioral logs and emotional data,

[0853] A means of presenting the generated automation proposal to the user and executing the approved action,

[0854] A means of adjusting proposals based on emotional state,

[0855] A system that includes this.

[0856] (Claim 2)

[0857] The system according to claim 1, further comprising means for adaptively updating the generative model based on feedback from users.

[0858] (Claim 3)

[0859] The system according to claim 1, further comprising means for encrypting and communicating the behavioral log and emotional data in order to enhance security.

[0860] "Application example 2 when combining with an emotional engine"

[0861] (Claim 1)

[0862] A means for aggregating a series of operations on the user's computing device,

[0863] A means for analyzing aggregated operation information and repetitive operations,

[0864] A means of creating automation suggestions for recognized repetitive tasks,

[0865] A means for performing learning using the aforementioned operational information and constructing a predictive model specialized for the business,

[0866] A means of presenting automation proposals to users and executing approved automations,

[0867] A means for detecting the worker's movement speed and voice, and for evaluating their emotional state,

[0868] Means for adjusting or proposing the work environment based on the aforementioned evaluation,

[0869] A system that includes this.

[0870] (Claim 2)

[0871] The system according to claim 1, further comprising means for updating the predictive model based on user feedback.

[0872] (Claim 3)

[0873] The system according to claim 1, further comprising means for encrypting the operation information and communicating it, with security in mind. [Explanation of symbols]

[0874] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means for collecting a series of operations on the user's computer device, A means for analyzing collected operational data and identifying repetitive operations, A means for generating automation suggestions for identified repetitive tasks, A means for performing learning using the aforementioned operational data and constructing a generative model specialized for the business, A means of presenting automation proposals to users and executing approved automations, A system that includes this.

2. The system according to claim 1, further comprising means for updating the generation model based on feedback from the user.

3. The system according to claim 1, further comprising means for encrypting the operation data and communicating it, with security in mind.

Images