system

Abstract

We provide the system. [Solution] Means for monitoring and collecting data related to user actions, A means for transmitting the collected data to a server and storing it in a database, Means for analyzing the stored data and identifying repetitive tasks, A means of generating scripts to automate identified repetitive tasks, A means for presenting the aforementioned script to the user, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern business environments, many businesspersons spend a lot of time on routine work, leading to a decline in productivity. Also, due to the lack of programming knowledge, they feel it difficult to introduce automation technologies by themselves. In such a situation, means for supporting the efficiency improvement of business operations are needed.

Means for Solving the Problems

[0005] This invention utilizes means for monitoring and collecting data related to user operations, and means for transmitting the collected data to a server and storing it in a database. Furthermore, it includes means for analyzing the stored data to identify repetitive tasks, and means for generating and presenting scripts to automate the identified tasks. This aims to automate and optimize user tasks, thereby improving operational efficiency. In addition, it provides comprehensive business support by including means for collecting location information and making efficiency suggestions based on movement history, as well as means for automatically generating and providing daily reports to the user.

[0006] "User" refers to an individual or organization that uses the system.

[0007] "Data related to operations" refers to information about a series of operations performed by a user on a device, including keyboard input, mouse operations, and application usage.

[0008] A "server" refers to a computer system used to receive, store, and analyze collected data.

[0009] A "database" refers to a digital repository of information used to systematically manage and store collected data.

[0010] "Analysis" refers to the process of using collected data to find patterns and characteristics.

[0011] "Repetitive tasks" refer to similar operations or actions that users perform regularly or repeatedly.

[0012] An "automation script" refers to a program that contains a set of commands or instructions designed to perform a specific task automatically.

[0013] "Location information" refers to data about a user's geographical location obtained using a smartphone or other device.

[0014] The "movement history" refers to data that records the changes in the user's location information over time.

[0015] The "efficiency improvement proposal" refers to specific guidelines and methods for improving the quality and speed of work.

[0016] The "daily report" refers to a report summarizing the work and activities carried out by the user in a day.

Brief Explanation of Drawings

[0017] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0018] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0019] First, the language used in the following description will be explained.

[0020] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0021] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0022] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0023] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0025] [First Embodiment]

[0026] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0027] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0030] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0033] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0037] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0038] This invention provides an embodiment of an AI system that streamlines operations and actions performed by users on a daily basis, thereby improving productivity. This system consists of a terminal, a server, and a program for coordinating them.

[0039] At system startup, the terminal monitors data related to user actions. This includes information such as keyboard and mouse usage and changes in the active window, and can be monitored in real time. Furthermore, the terminal uses the smartphone's location information to record the user's movement history. This information forms the basis for understanding the user's daily behavior.

[0040] Data collected by the device is periodically encrypted and securely transmitted to the server. The server stores this data in a database and manages it with appropriate tags. After the data is stored, advanced analytical processing is performed on the server. This analysis identifies recurring work patterns of the user and routes that can be optimized for movement.

[0041] Based on the analysis, the server generates automation scenarios and improvement suggestions to help users improve their work efficiency. This includes scripts to automate routine tasks performed on a daily basis. The server presents these suggestions and scripts to the user and notifies them through applications and email.

[0042] As a concrete example, if a user performs the task of entering specific data into a spreadsheet every morning, the terminal records this data, and the server generates a script to simplify the task. This script allows the user to complete the task with a single button click, without having to repeat the same task each time. Furthermore, the server analyzes the user's travel history and suggests the shortest route and reduces travel time, enabling them to dedicate wasted time to more creative activities.

[0043] Based on the user's daily activities, the server automatically generates daily reports, supporting the user in understanding the overall picture of streamlined operations. In this way, the present invention provides comprehensive support to reduce the user's workload and improve the creativity and efficiency of their work.

[0044] The following describes the processing flow.

[0045] Step 1:

[0046] The terminal monitors and collects data related to user actions in real time. This is done by recording information such as keyboard input, mouse clicks, and changes in window focus.

[0047] Step 2:

[0048] The device periodically acquires location information from the user's smartphone and records their movement history. This allows the system to understand the user's daily movement patterns.

[0049] Step 3:

[0050] The device packages all collected data at regular intervals and encrypts it for security purposes.

[0051] Step 4:

[0052] The terminal sends an encrypted data package to the server via the internet. A confirmation of transmission is performed to ensure the reliability of the data transfer.

[0053] Step 5:

[0054] The server decompresses the received data and saves it to the database in the appropriate format. The data is categorized and managed as application usage history, location information, and operation records.

[0055] Step 6:

[0056] The server analyzes the stored data. Specifically, it uses machine learning algorithms to detect specific patterns and repetitive tasks. The insights gained from this analysis are stored in a database.

[0057] Step 7:

[0058] Based on the analysis results, the server identifies areas where business processes can be streamlined and generates automation scripts. These scripts are intended to automate the user's own tasks.

[0059] Step 8:

[0060] The server notifies the user of generated scripts and efficiency suggestions. Notifications are sent via a dedicated application or email.

[0061] Step 9:

[0062] Users review the received proposals and scripts and incorporate them into their workflow as needed. Once the user approves, the proposed automation script is executed.

[0063] Step 10:

[0064] The server automatically generates and provides daily reports to users based on their activity history. These reports include information on work progress and suggestions for improvement.

[0065] (Example 1)

[0066] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0067] In today's technological environment, users routinely perform many repetitive tasks and movements, which require significant time and effort, posing a challenge to productivity. To streamline and automate these daily tasks, technologies are needed that accurately capture, analyze, and propose solutions for these work patterns and movements. However, current methods lack the means to handle these tasks comprehensively and safely.

[0068] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0069] In this invention, the server includes means for acquiring information related to user operations, means for transferring the acquired information to a remote storage device and storing it in a data set, and means for analyzing the stored information and recognizing repetitive tasks. This enables the user to automate daily repetitive tasks, improve travel efficiency, and save time and effort.

[0070] "Information related to user actions" refers to data about user actions on a computer or terminal, such as keyboard input, mouse clicks, and changes in the active window.

[0071] A "remote storage device" refers to a storage system accessible via a network, such as a server or a cloud-based database, for storing collected data.

[0072] A "data set" is a collection of data that has been gathered and stored, managed together, and used for later analysis and processing.

[0073] "Repetitive tasks" refer to similar operations or tasks that users perform many times on a daily basis, and identifying these tasks makes them suitable for automation.

[0074] "Processing code" refers to scripts or programs generated to automate specific repetitive tasks, serving as a means to simplify the user's work.

[0075] "Suggestions for improving travel efficiency" are specific advice to eliminate waste by analyzing the user's travel history to determine the optimal travel route and time.

[0076] This invention is a system for streamlining user operations and movement, and consists of a terminal, a server, and a program to link them together. In this system, the terminal monitors information related to user operations in real time. Specifically, it uses dedicated monitoring software to log keyboard input, mouse operations, and window activity status. In addition, a smartphone is used as a terminal, and location information services are utilized to continuously acquire the user's movement history.

[0077] Information collected by the device is encrypted to ensure data security. Standard encryption algorithms such as AES-256 are used for encryption. The encrypted information is then periodically sent to the server via the secure HTTPS protocol. The server stores the received information in a database and tags it with metadata such as date and data type.

[0078] The server analyzes stored information using a generated AI model. This analysis identifies recurring work patterns and travel routes of the user. Based on the identified work patterns, the server generates processing code to automate the user's routine tasks. The generated processing code is a means to simplify and automate the user's work. In addition, the server analyzes the user's travel history and makes suggestions for more efficient travel.

[0079] For example, if a user enters data into a spreadsheet from a specific website every morning, the terminal records this work pattern. Based on this, the server generates processing code to automate the data entry process and provides it to the user. This prompt allows the user to complete the task with a single click, improving productivity. Additionally, the system suggests optimal routes through analysis of travel history, contributing to shorter commute times and reduced unnecessary travel.

[0080] An example of a prompt message is, "Create an automated spreadsheet input scenario based on the user's work patterns over the past week." This structure allows users to enjoy increased efficiency in their daily tasks.

[0081] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0082] Step 1:

[0083] Data collection by devices

[0084] The terminal monitors user activity in real time and collects data. Specifically, it uses monitoring software to acquire keyboard input, mouse movements, and window activity status. Input is user activity events, and output is a log file containing this data. In this step, the terminal continuously records the user's daily activities.

[0085] Step 2:

[0086] Location information collection by the device

[0087] The device uses the smartphone's location information to collect the user's movement history. It utilizes GPS data and Wi-Fi location information to record the user's current location and travel route. The input is the user's location data, and the output is a collection of data points representing the movement history. This allows the device to create a foundation for understanding the user's movement patterns.

[0088] Step 3:

[0089] Data encryption and transmission

[0090] The device encrypts the collected data and sends it to the server. Encryption algorithms such as AES-256 are used to secure the data. The input is the previously collected operation and location data, and the output is an encrypted data package. This data is sent to the server via the HTTPS protocol.

[0091] Step 4:

[0092] Server-based data storage and tagging

[0093] The server receives encrypted data sent from the terminal and stores it in a database. The data is tagged with metadata such as date and type, preparing it for subsequent processing. The input is an encrypted data package, and the output is a tagged database entry. This enables data organization and efficient management.

[0094] Step 5:

[0095] Server-based data analysis

[0096] The server uses a generative AI model based on stored data to recognize the user's repetitive tasks and travel routes. The input is stored database entries, and the output is recognized work patterns and optimization suggestions. This allows the server to present the user with necessary improvement suggestions.

[0097] Step 6:

[0098] Generating automation scripts

[0099] The server generates automation scripts for identified repetitive tasks. It leverages a generative AI model to create code that streamlines user operations. The input is a recognized work pattern, and the output is an automation script. The server uses this script to simplify the user's tasks.

[0100] Step 7:

[0101] Improvement suggestions and script notifications

[0102] The server notifies the user of the generated scripts and suggestions for improving travel efficiency. The input is the generated scripts and suggestions, and the output is a notification message to the user. This allows the user to improve the efficiency of their work based on the suggestions.

[0103] (Application Example 1)

[0104] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0105] In factories, robot operation and route selection often rely on experience and intuition, making efficient operation difficult. Furthermore, repetitive tasks and unnecessary movements in daily operations can reduce productivity and lead to a lack of optimization, which needs to be addressed.

[0106] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0107] In this invention, the server includes means for monitoring and collecting data related to user operations, means for transmitting the collected data to a communication device and storing it in a storage device, means for analyzing the stored data to identify repetitive tasks, means for generating instructions to automate the identified repetitive tasks, means for analyzing the robot's operation history to identify optimized operation patterns, and means for generating instructions to optimize the operation path based on the identified operation patterns. This significantly improves the operational efficiency of robots in a factory, enabling reduced working time and increased productivity.

[0108] "Data related to user actions" refers to all information related to user actions on a computer or device, such as keyboard input, mouse operations, and changes to the active window.

[0109] "Means for transmitting collected data to a communication device and storing it in a storage device" refers to technical methods and devices for securely transmitting and storing data collected by a terminal device to a storage device such as a server via a network.

[0110] "Means for identifying repetitive tasks" refers to methods and devices for analyzing stored data to detect and identify routine tasks that users perform on a daily basis.

[0111] "Instructions for automation" refers to the content of instructions for generating specific operational procedures or scripts to automate and streamline identified tasks.

[0112] "Robot operation history" refers to a record of actions and route selections performed by a robot in the past, and is information used to improve work efficiency.

[0113] "Optimized motion patterns" refer to the optimized robot movements and route selection methods obtained as a result of analysis.

[0114] "Instructions for optimizing the movement path" refers to instructions given to a robot to select and guide it to the shortest route and a movement path that takes obstacles into consideration, in order to reach its destination efficiently.

[0115] The system for implementing this invention aims to efficiently collect and analyze user operation and activity history. The terminal monitors data related to user operations in real time and records information such as keyboard input and mouse operations. Furthermore, it utilizes location information to track the user's movement history and construct a detailed activity log.

[0116] The collected data is securely transmitted to the server via a communication device after undergoing an encryption process. The server stores the data in dedicated storage devices and then efficiently tags it in a database. The server then uses AI models and algorithms to analyze the data and automatically identify repetitive user tasks and optimize travel routes. To achieve this, the system utilizes sensors and GPS devices as hardware, and Python, AWS® SageMaker, and DynamoDB as software.

[0117] Based on the analysis results, the server generates specific instructions for automation and operational efficiency. For robots, it suggests the optimal movement path based on the optimized movement patterns and issues instructions to immediately execute those patterns. Finally, these instructions are delivered via communication to the user's mobile device or robot control system, thereby improving operational efficiency.

[0118] As a concrete example, in parts picking operations in a factory, it is possible to instruct robots on the shortest route and most efficient operating procedures by analyzing historical data accumulated by the user. This can significantly reduce picking time and improve productivity.

[0119] Examples of prompts for a generative AI model:

[0120] "Based on past movement data, please provide a detailed description of the most efficient route for parts picking. The goal is to complete the picking in the shortest possible time. Please also propose alternative routes in case of obstacles."

[0121] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0122] Step 1:

[0123] The terminal monitors user actions in real time, collecting information on keyboard and mouse operations, as well as the activity of the active window. This information is recorded as a user action dataset. The input is user action information, and the output is stored as a dataset.

[0124] Step 2:

[0125] The terminal encrypts the collected data using AES encryption and securely transmits it to the server via a communication device. This process ensures data security. The input is the unencrypted operation data, and the output is the encrypted data.

[0126] Step 3:

[0127] The server decrypts the received encrypted data and stores it in a database. Appropriate tagging is applied during storage to ensure efficient use in subsequent processing. The input is encrypted data, and the output is a tagged database entry.

[0128] Step 4:

[0129] The server uses stored data to leverage an AI model and perform data analysis. This identifies repetitive user tasks and discovers patterns that can be improved. The input is database entries, and the output is the identified improvement patterns.

[0130] Step 5:

[0131] Based on this analysis, the server generates instructions that can be automated and analyzes the robot's motion history to identify more efficient motion patterns. The instructions obtained through this process are compiled as a suggestion for the robot's optimal motion path. The input is the improved pattern, and the output is the content of the instructions.

[0132] Step 6:

[0133] The server notifies the robot control system and the user's device of the generated instructions. The user and robot then perform specific actions based on these instructions, improving productivity. The input is the generated instructions, and the output is the actual operational action.

[0134] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0135] This invention describes a configuration that provides more personalized support to users by combining an emotion engine with an AI system that analyzes and optimizes user operations and behavior. The system consists of a user terminal, a server, and a program that integrates the emotion engine.

[0136] At system startup, the terminal collects user activity and location information. This data includes information about the applications the user is running and location information obtained from the smartphone. The terminal encrypts this data and sends it to the server.

[0137] The server stores the received data in a database. The stored data is then analyzed in detail to identify repetitive operations, and suggestions for improving operational efficiency are created based on the results. In particular, scripts for automating repetitive tasks are generated and presented to the user.

[0138] The emotion engine recognizes the user's emotions and provides feedback tailored to the user's state based on that information. For example, the device analyzes the user's facial expressions and tone of voice through the camera and voice input to detect changes in emotion. The server takes in the data obtained from the emotion engine and analyzes the user's emotional history. This allows for new approaches to reduce the frequency of negative emotions and suggestions for improving work processes.

[0139] For example, if a user seems to be finding a task burdensome, the server might suggest automating that task, and the emotion engine might provide feedback such as recommending relaxing music. Furthermore, the system adjusts the content of the automation script in consideration of changes in the user's emotions to help them perform their work comfortably.

[0140] Thus, by considering both user behavior and emotions, the present invention achieves comprehensive user support that goes beyond simply improving work efficiency, and helps users work in a more comfortable and productive environment.

[0141] The following describes the processing flow.

[0142] Step 1:

[0143] The device monitors user activity and collects application usage and keyboard and mouse input information. It also obtains the smartphone's location information and records the user's daily movement patterns.

[0144] Step 2:

[0145] The device uses its camera and microphone to acquire data in real time, including the user's facial expressions and voice tone, which are then input into the emotion engine. Based on this data, the system determines the user's emotional state.

[0146] Step 3:

[0147] The device integrates and packages collected operational data, location information, and emotional data, and securely transmits it to the server. During this process, the data is encrypted to ensure security.

[0148] Step 4:

[0149] The server decompresses the received data, categorizes it according to its relevant category, and stores it in the database. This data is managed as operation history, movement history, and sentiment history.

[0150] Step 5:

[0151] The server analyzes the stored operation data to identify patterns in repetitive tasks. Based on these patterns, it identifies parts that can be automated and generates scripts.

[0152] Step 6:

[0153] The server analyzes data from the emotion engine to investigate the user's emotional history. This analysis identifies tasks and times of day when the user is likely to experience stress.

[0154] Step 7:

[0155] The server generates automation scripts for identified repetitive tasks and suggestions for improving operations based on sentiment analysis, and notifies the user. These notifications include reminders and specific actions.

[0156] Step 8:

[0157] The user reviews the suggestions and scripts from the server and approves or adjusts their execution as needed. If the user approves, the script is applied and repetitive tasks are automated.

[0158] Step 9:

[0159] The server collects user behavior and emotional history and automatically generates daily reports. These reports include feedback for business improvement and suggestions based on the user's emotional state.

[0160] Step 10:

[0161] Through the daily reports provided, users gain insights into their own work efficiency and emotional management. This allows them to explore even more efficient ways of working.

[0162] (Example 2)

[0163] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0164] Conventional systems were limited to automating repetitive tasks based on user input and suggesting improvements to operational efficiency, but they failed to provide feedback that took into account the user's emotional state. As a result, even if operational efficiency improved, the user's mental stress and dissatisfaction remained unresolved.

[0165] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0166] In this invention, the server includes means for collecting and storing information related to user operations, means for identifying and generating procedures for automating repetitive tasks, and means for analyzing the user's emotional state and providing feedback. This makes it possible to not only improve work efficiency but also to consider the user's emotional state, thereby reducing stress and providing a comfortable work environment.

[0167] "Information related to user operations" refers to data generated when a user uses electronic devices or applications, and includes operation history and usage time.

[0168] A "computer device" is a device equipped with hardware and software for data processing, and is primarily used as a server.

[0169] A "storage device" is a device that has the function of saving data, and includes hard disks and SSDs.

[0170] "Means for identifying repetitive tasks" refers to a function that uses data analysis techniques to find patterns and routines from user operation data and extracts repetitive tasks as tasks that can be automated.

[0171] "Means of generating procedures" refers to the ability to assemble the operations and processes necessary to streamline or automate identified repetitive tasks.

[0172] "Means for analyzing emotional states" refers to technologies that detect emotions from a user's facial expressions, voice, etc., and evaluate that state.

[0173] "Means of providing feedback" refers to a function that presents users with advice and improvement suggestions tailored to their current state and work content.

[0174] "Proposals for improving business efficiency" involve presenting more efficient and rational operating methods and improvement measures based on an analysis of the user's work and processes.

[0175] "Methods for automatically generating reports" refers to a function that automatically creates daily reports, monthly reports, etc., in a specified format based on the user's work history data.

[0176] This invention is a system that comprehensively analyzes user operation data and emotional information to achieve both improved work efficiency and emotional support for users. Specific embodiments are described below.

[0177] First, the device collects user activity data. This includes information such as smartphone and personal computer usage history and application launch times. This data is encrypted using the SSL / TLS protocol within the device and then sent to the server.

[0178] The server stores the received data in a relational database. This data is analyzed using data mining techniques to identify recurring operation patterns. Based on the results of this analysis, procedures for automating repetitive tasks are generated and presented to the user.

[0179] Simultaneously, the device uses its camera and microphone to analyze the user's emotional state in real time. The emotion engine analyzes facial expressions and tone of voice from this data to evaluate how the user is feeling about their current task.

[0180] The server provides feedback to the user based on the analysis results of the emotion engine. For example, if the user is feeling stressed about their work, it may offer relaxing music or suggest automation to reduce their workload.

[0181] For example, if a user is feeling burdened by a particular task, the server will generate an automation script for that task and present it to the user. Furthermore, as feedback that takes the user's emotional state into consideration, relaxing music may be recommended.

[0182] As an example of using a generative AI model, here is an example of a prompt: "Please suggest a script to automate a specific operation that the user finds stressful, and recommend relaxing music."

[0183] Thus, the invention comprehensively utilizes user operation data and emotional information to improve not only work efficiency but also user comfort and productivity.

[0184] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0185] Step 1:

[0186] The terminal collects user operation data. Specifically, the terminal monitors the user's application usage history and operation time in real time and records it in a database. The input is user operation information, and the output is stored in the format of date, time, application name, and usage time.

[0187] Step 2:

[0188] The device collects the user's location information. Using its built-in GPS sensor, the device determines the user's movement path and current location. Input is GPS sensor data, and output is geographical coordinate information and movement history. This data is temporarily stored within the device.

[0189] Step 3:

[0190] The device encrypts the collected data and sends it to the server. Specifically, it securely encrypts the data using the SSL / TLS protocol and sends it to the server over the internet. The input is collected operation data and location information, and the output is an encrypted data stream.

[0191] Step 4:

[0192] The server stores the received data in a database. The server verifies the incoming data in real time, confirms its integrity, and then stores it in the relational database. The input is encrypted data sent from the terminal, and the output is organized data entries in the database.

[0193] Step 5:

[0194] The server analyzes the stored data and identifies recurring operations. Here, a data analysis algorithm is used to identify frequently used operations. The input is the operation history in the database, and the output is a list of frequent patterns. This result reveals which tasks can be made more efficient.

[0195] Step 6:

[0196] The server generates and presents to the user a script to automate the identified repetitive tasks. The server uses a generation AI model to automatically generate code that automates the repetitive tasks. The input is the repetitive patterns from the analysis results, and the output is an automation script in a user-friendly format.

[0197] Step 7:

[0198] The device analyzes the user's emotional state. The device uses a camera and microphone to collect the user's facial expressions and voice, and analyzes them using an emotion engine. The input is video and audio data, and the output is metadata indicating the user's emotional state.

[0199] Step 8:

[0200] The server provides feedback based on data from the emotion engine. Based on the emotion analysis results, the server recommends music to reduce stress and suggests improvements to work processes. The input is data on the user's emotional state, and the output provides the user with recommended music and specific action plans.

[0201] (Application Example 2)

[0202] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0203] In modern work environments, there is a widespread demand for automation to improve user efficiency. However, conventional systems often automate and streamline tasks without considering the user's emotional state, which can actually increase stress. Therefore, it is necessary to improve work efficiency while simultaneously reducing the user's mental burden and providing a comfortable work environment.

[0204] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0205] In this invention, the server includes means for monitoring and collecting data related to the user's actions and emotional state; means for encrypting the collected data and transmitting it to a data management server and storing it in a data storage device; and means for analyzing the stored data and identifying repetitive tasks and emotional changes. This makes it possible to adjust the user's workload and provide encouraging information, thereby improving work efficiency and reducing mental burden.

[0206] "User actions" refer to a series of actions or instructions that a user performs on a device or system.

[0207] "Emotional state" refers to information that indicates the user's psychological state, derived from their facial expressions and voice.

[0208] A "data management server" is an external server that processes and stores information related to user actions and emotions.

[0209] A "data storage device" is a storage medium for stably and securely storing collected data.

[0210] "Saving" refers to the process of ensuring that acquired data is not lost.

[0211] "Analysis" refers to the process of using collected data to derive trends and characteristics.

[0212] "Repetitive tasks" refer to similar actions or processes that users frequently perform on the system.

[0213] "Emotional changes" refer to how a user's emotions evolve over time.

[0214] A "processing procedure" refers to a series of automated steps designed to streamline repetitive tasks.

[0215] "Information" refers to suggestions based on the user's emotional state, aimed at adjusting workload and providing encouragement.

[0216] The system for realizing this invention aims to adjust the user's workload and reduce mental burden by collecting data related to the user's actions and emotional state and performing efficient information processing. The system mainly consists of a terminal, a data management server, an emotion analysis engine, and a data storage device.

[0217] The device is responsible for capturing the user's emotional state through user interaction and the use of a camera and microphone. The emotion analysis engine uses this data to determine the user's emotions in real time and record changes in those emotions. Face detection using OpenCV and speech analysis using TENSORFLOW® are performed.

[0218] The data management server encrypts and securely stores the collected data. The Python Cryptography library is used, and data storage is performed by a data storage device. The server also performs detailed analysis using the stored data to identify repetitive tasks and changes in sentiment. This involves data analysis using Python libraries such as Pandas and NumPy, generating user-friendly workflows.

[0219] For example, if the emotion analysis engine determines that a user is experiencing stress during a particular task, the server uses this information to appropriately adjust the workload and generate and display an encouraging message on the terminal. The system uses a generative AI model and employs this prompt: "Based on input data from the camera and microphone, determine the worker's emotional state. If signs of fatigue or stress are detected, recommend an appropriate encouraging message and temporary task automation."

[0220] In this way, it is possible to optimize the user's work environment, increase productivity, and continue to support their mental health.

[0221] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0222] Step 1:

[0223] The device collects the user's emotional state using user actions and its camera and microphone. This utilizes sensors and input devices on the device. The input data includes the user's behavioral patterns, facial expressions, and voice tone. This input data is converted into an analyzable format through signal processing and digital conversion.

[0224] Step 2:

[0225] The terminal encrypts the collected data and sends it to the data management server in a secure state. The Python Cryptography library is used for encryption. This ensures that the user's personal information is protected when the data is sent to the server. The server receives this data and stores it in a data storage device.

[0226] Step 3:

[0227] The server analyzes the stored data using a data management system. Here, Python's Pandas and NumPy are used to analyze the data and identify patterns in repetitive tasks and emotional changes. The stored data is processed using the aforementioned tools for data analysis, and the output provides characteristic information about repetitive tasks and emotional changes.

[0228] Step 4:

[0229] The server generates information for adjusting workload and providing emotional feedback based on identified task characteristics and emotional change data. This includes a process that uses a generative AI model and prompt statements to generate appropriate responses and suggestions for task automation. This generated information is then sent back to the terminal.

[0230] Step 5:

[0231] Based on information received from the server, the terminal suggests adjusting the user's workload and displays encouraging messages. For example, it might send specific messages like "Let's take a short break" or notifications such as "An automated task will begin." As output, the suggestions are reflected on the user's screen, providing feedback to the user's actions.

[0232] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0233] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0234] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0235] [Second Embodiment]

[0236] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0237] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0238] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0239] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0240] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0241] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0242] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0243] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0244] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0245] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0246] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0247] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0248] This invention provides an embodiment of an AI system that streamlines operations and actions performed by users on a daily basis, thereby improving productivity. This system consists of a terminal, a server, and a program for coordinating them.

[0249] At system startup, the terminal monitors data related to user actions. This includes information such as keyboard and mouse usage and changes in the active window, and can be monitored in real time. Furthermore, the terminal uses the smartphone's location information to record the user's movement history. This information forms the basis for understanding the user's daily behavior.

[0250] Data collected by the device is periodically encrypted and securely transmitted to the server. The server stores this data in a database and manages it with appropriate tags. After the data is stored, advanced analytical processing is performed on the server. This analysis identifies recurring work patterns of the user and routes that can be optimized for movement.

[0251] Based on the analysis, the server generates automation scenarios and improvement suggestions to help users improve their work efficiency. This includes scripts to automate routine tasks performed on a daily basis. The server presents these suggestions and scripts to the user and notifies them through applications and email.

[0252] As a concrete example, if a user performs the task of entering specific data into a spreadsheet every morning, the terminal records this data, and the server generates a script to simplify the task. This script allows the user to complete the task with a single button click, without having to repeat the same task each time. Furthermore, the server analyzes the user's travel history and suggests the shortest route and reduces travel time, enabling them to dedicate wasted time to more creative activities.

[0253] Based on the user's daily activities, the server automatically generates daily reports, supporting the user in understanding the overall picture of streamlined operations. In this way, the present invention provides comprehensive support to reduce the user's workload and improve the creativity and efficiency of their work.

[0254] The following describes the processing flow.

[0255] Step 1:

[0256] The terminal monitors and collects data related to user actions in real time. This is done by recording information such as keyboard input, mouse clicks, and changes in window focus.

[0257] Step 2:

[0258] The device periodically acquires location information from the user's smartphone and records their movement history. This allows the system to understand the user's daily movement patterns.

[0259] Step 3:

[0260] The device packages all collected data at regular intervals and encrypts it for security purposes.

[0261] Step 4:

[0262] The terminal sends an encrypted data package to the server via the internet. A confirmation of transmission is performed to ensure the reliability of the data transfer.

[0263] Step 5:

[0264] The server decompresses the received data and saves it to the database in the appropriate format. The data is categorized and managed as application usage history, location information, and operation records.

[0265] Step 6:

[0266] The server analyzes the stored data. Specifically, it uses machine learning algorithms to detect specific patterns and repetitive tasks. The insights gained from this analysis are stored in a database.

[0267] Step 7:

[0268] Based on the analysis results, the server identifies areas where business processes can be streamlined and generates automation scripts. These scripts are intended to automate the user's own tasks.

[0269] Step 8:

[0270] The server notifies the user of generated scripts and efficiency suggestions. Notifications are sent via a dedicated application or email.

[0271] Step 9:

[0272] Users review the received proposals and scripts and incorporate them into their workflow as needed. Once the user approves, the proposed automation script is executed.

[0273] Step 10:

[0274] The server automatically generates and provides daily reports to users based on their activity history. These reports include information on work progress and suggestions for improvement.

[0275] (Example 1)

[0276] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0277] In today's technological environment, users routinely perform many repetitive tasks and movements, which require significant time and effort, posing a challenge to productivity. To streamline and automate these daily tasks, technologies are needed that accurately capture, analyze, and propose solutions for these work patterns and movements. However, current methods lack the means to handle these tasks comprehensively and safely.

[0278] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0279] In this invention, the server includes means for acquiring information related to user operations, means for transferring the acquired information to a remote storage device and storing it in a data set, and means for analyzing the stored information and recognizing repetitive tasks. This enables the user to automate daily repetitive tasks, improve travel efficiency, and save time and effort.

[0280] "Information related to user operations" refers to data related to user actions such as keyboard input, mouse clicks, and active window changes performed by the user on a computer or terminal.

[0281] "Remote storage device" refers to a storage system accessible via a network, such as a server or a database on the cloud for storing the collected data.

[0282] "Data set" refers to a collection of data that manages the collected and stored data collectively for later analysis and processing.

[0283] "Repetitive work" refers to similar operations or tasks that the user performs daily many times, and by identifying this, it is the work targeted for automation.

[0284] "Processing code" refers to a script or program generated to automate a specific repetitive work and is a means to simplify the user's work.

[0285] "Proposal for improving movement efficiency" refers to specific advice for deriving an optimal movement route and time and saving waste as a result of analyzing the user's movement history.

[0286] The present invention is a system for improving the efficiency of user operations and movements, and is composed of a terminal, a server, and a program for coordinating them. In this system, the terminal monitors information related to user operations in real time. Specifically, using dedicated monitoring software, it records keyboard input, mouse operations, and the active state of the window as logs. Also, using a smartphone as the terminal, it continuously obtains the user's movement history by utilizing the location information service.

[0287] Information collected by the device is encrypted to ensure data security. Standard encryption algorithms such as AES-256 are used for encryption. The encrypted information is then periodically sent to the server via the secure HTTPS protocol. The server stores the received information in a database and tags it with metadata such as date and data type.

[0288] The server analyzes stored information using a generated AI model. This analysis identifies recurring work patterns and travel routes of the user. Based on the identified work patterns, the server generates processing code to automate the user's routine tasks. The generated processing code is a means to simplify and automate the user's work. In addition, the server analyzes the user's travel history and makes suggestions for more efficient travel.

[0289] For example, if a user enters data into a spreadsheet from a specific website every morning, the terminal records this work pattern. Based on this, the server generates processing code to automate the data entry process and provides it to the user. This prompt allows the user to complete the task with a single click, improving productivity. Additionally, the system suggests optimal routes through analysis of travel history, contributing to shorter commute times and reduced unnecessary travel.

[0290] An example of a prompt message is, "Create an automated spreadsheet input scenario based on the user's work patterns over the past week." This structure allows users to enjoy increased efficiency in their daily tasks.

[0291] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0292] Step 1:

[0293] Data collection by devices

[0294] The terminal monitors user activity in real time and collects data. Specifically, it uses monitoring software to acquire keyboard input, mouse movements, and window activity status. Input is user activity events, and output is a log file containing this data. In this step, the terminal continuously records the user's daily activities.

[0295] Step 2:

[0296] Location information collection by the device

[0297] The device uses the smartphone's location information to collect the user's movement history. It utilizes GPS data and Wi-Fi location information to record the user's current location and travel route. The input is the user's location data, and the output is a collection of data points representing the movement history. This allows the device to create a foundation for understanding the user's movement patterns.

[0298] Step 3:

[0299] Data encryption and transmission

[0300] The device encrypts the collected data and sends it to the server. Encryption algorithms such as AES-256 are used to secure the data. The input is the previously collected operation and location data, and the output is an encrypted data package. This data is sent to the server via the HTTPS protocol.

[0301] Step 4:

[0302] Server-based data storage and tagging

[0303] The server receives encrypted data sent from the terminal and stores it in a database. The data is tagged with metadata such as date and type, preparing it for subsequent processing. The input is an encrypted data package, and the output is a tagged database entry. This enables data organization and efficient management.

[0304] Step 5:

[0305] Data analysis by the server

[0306] Based on the stored data, the server uses a generative AI model to recognize the user's repetitive tasks and movement routes. The input is the stored database entry, and the output is the recognized work pattern and optimization suggestions. This enables the server to present improvement proposals required by the user.

[0307] Step 6:

[0308] Generation of automation scripts

[0309] <> The server generates an automation script for the identified repetitive tasks. Utilizing the generative AI model, it creates code to streamline the user's operations. The input is the recognized work pattern, and the output is the automation script. The server simplifies the user's work using this script. [[END4]]

[0310] Step 7:

[0311] Notification of improvement proposals and scripts

[0312] The server notifies the user of the generated script and proposals for movement efficiency. The input is the generated script and proposal content, and the output is the notification message to the user. This allows the user to improve the efficiency of their work based on the proposals.

[0313] (Application Example 1)

[0314] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0315] In factories, robot operation and route selection often rely on experience and intuition, making efficient operation difficult. Furthermore, repetitive tasks and unnecessary movements in daily operations can reduce productivity and lead to a lack of optimization, which needs to be addressed.

[0316] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0317] In this invention, the server includes means for monitoring and collecting data related to user operations, means for transmitting the collected data to a communication device and storing it in a storage device, means for analyzing the stored data to identify repetitive tasks, means for generating instructions to automate the identified repetitive tasks, means for analyzing the robot's operation history to identify optimized operation patterns, and means for generating instructions to optimize the operation path based on the identified operation patterns. This significantly improves the operational efficiency of robots in a factory, enabling reduced working time and increased productivity.

[0318] "Data related to user actions" refers to all information related to user actions on a computer or device, such as keyboard input, mouse operations, and changes to the active window.

[0319] "Means for transmitting collected data to a communication device and storing it in a storage device" refers to technical methods and devices for securely transmitting and storing data collected by a terminal device to a storage device such as a server via a network.

[0320] "Means for identifying repetitive tasks" refers to methods and devices for analyzing stored data to detect and identify routine tasks that users perform on a daily basis.

[0321] "Instructions for automation" refers to the content of instructions for generating specific operational procedures or scripts to automate and streamline identified tasks.

[0322] "Robot operation history" refers to a record of actions and route selections performed by a robot in the past, and is information used to improve work efficiency.

[0323] "Optimized motion patterns" refer to the optimized robot movements and route selection methods obtained as a result of analysis.

[0324] "Instructions for optimizing the movement path" refers to instructions given to a robot to select and guide it to the shortest route and a movement path that takes obstacles into consideration, in order to reach its destination efficiently.

[0325] The system for implementing this invention aims to efficiently collect and analyze user operation and activity history. The terminal monitors data related to user operations in real time and records information such as keyboard input and mouse operations. Furthermore, it utilizes location information to track the user's movement history and construct a detailed activity log.

[0326] The collected data is securely transmitted to the server via a communication device after undergoing an encryption process. The server stores the data in dedicated storage devices and then efficiently tags it in a database. The server then uses AI models and algorithms to analyze the data and automatically identify repetitive user tasks and optimize travel routes. To achieve this, the system utilizes sensors and GPS devices as hardware, and Python, AWS SageMaker, and DynamoDB as software.

[0327] Based on the analysis results, the server generates specific instructions for automation and operational efficiency. For robots, it suggests the optimal movement path based on the optimized movement patterns and issues instructions to immediately execute those patterns. Finally, these instructions are delivered via communication to the user's mobile device or robot control system, thereby improving operational efficiency.

[0328] As a concrete example, in parts picking operations in a factory, it is possible to instruct robots on the shortest route and most efficient operating procedures by analyzing historical data accumulated by the user. This can significantly reduce picking time and improve productivity.

[0329] Examples of prompts for a generative AI model:

[0330] "Based on past movement data, please provide a detailed description of the most efficient route for parts picking. The goal is to complete the picking in the shortest possible time. Please also propose alternative routes in case of obstacles."

[0331] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0332] Step 1:

[0333] The terminal monitors user actions in real time, collecting information on keyboard and mouse operations, as well as the activity of the active window. This information is recorded as a user action dataset. The input is user action information, and the output is stored as a dataset.

[0334] Step 2:

[0335] The terminal encrypts the collected data using AES encryption and securely transmits it to the server via a communication device. This process ensures data security. The input is the unencrypted operation data, and the output is the encrypted data.

[0336] Step 3:

[0337] The server decrypts the received encrypted data and stores it in a database. Appropriate tagging is applied during storage to ensure efficient use in subsequent processing. The input is encrypted data, and the output is a tagged database entry.

[0338] Step 4:

[0339] The server uses stored data to leverage an AI model and perform data analysis. This identifies repetitive user tasks and discovers patterns that can be improved. The input is database entries, and the output is the identified improvement patterns.

[0340] Step 5:

[0341] Based on this analysis, the server generates instructions that can be automated and analyzes the robot's motion history to identify more efficient motion patterns. The instructions obtained through this process are compiled as a suggestion for the robot's optimal motion path. The input is the improved pattern, and the output is the content of the instructions.

[0342] Step 6:

[0343] The server notifies the robot control system and the user's device of the generated instructions. The user and robot then perform specific actions based on these instructions, improving productivity. The input is the generated instructions, and the output is the actual operational action.

[0344] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0345] This invention describes a configuration that provides more personalized support to users by combining an emotion engine with an AI system that analyzes and optimizes user operations and behavior. The system consists of a user terminal, a server, and a program that integrates the emotion engine.

[0346] At system startup, the terminal collects user activity and location information. This data includes information about the applications the user is running and location information obtained from the smartphone. The terminal encrypts this data and sends it to the server.

[0347] The server stores the received data in a database. The stored data is then analyzed in detail to identify repetitive operations, and suggestions for improving operational efficiency are created based on the results. In particular, scripts for automating repetitive tasks are generated and presented to the user.

[0348] The emotion engine recognizes the user's emotions and provides feedback tailored to the user's state based on that information. For example, the device analyzes the user's facial expressions and tone of voice through the camera and voice input to detect changes in emotion. The server takes in the data obtained from the emotion engine and analyzes the user's emotional history. This allows for new approaches to reduce the frequency of negative emotions and suggestions for improving work processes.

[0349] For example, if a user seems to be finding a task burdensome, the server might suggest automating that task, and the emotion engine might provide feedback such as recommending relaxing music. Furthermore, the system adjusts the content of the automation script in consideration of changes in the user's emotions to help them perform their work comfortably.

[0350] Thus, by considering both user behavior and emotions, the present invention achieves comprehensive user support that goes beyond simply improving work efficiency, and helps users work in a more comfortable and productive environment.

[0351] The following describes the processing flow.

[0352] Step 1:

[0353] The device monitors user activity and collects application usage and keyboard and mouse input information. It also obtains the smartphone's location information and records the user's daily movement patterns.

[0354] Step 2:

[0355] The device uses its camera and microphone to acquire data in real time, including the user's facial expressions and voice tone, which are then input into the emotion engine. Based on this data, the system determines the user's emotional state.

[0356] Step 3:

[0357] The device integrates and packages collected operational data, location information, and emotional data, and securely transmits it to the server. During this process, the data is encrypted to ensure security.

[0358] Step 4:

[0359] The server decompresses the received data, categorizes it according to its relevant category, and stores it in the database. This data is managed as operation history, movement history, and sentiment history.

[0360] Step 5:

[0361] The server analyzes the stored operation data to identify patterns in repetitive tasks. Based on these patterns, it identifies parts that can be automated and generates scripts.

[0362] Step 6:

[0363] The server analyzes data from the emotion engine to investigate the user's emotional history. This analysis identifies tasks and times of day when the user is likely to experience stress.

[0364] Step 7:

[0365] The server generates automation scripts for identified repetitive tasks and suggestions for improving operations based on sentiment analysis, and notifies the user. These notifications include reminders and specific actions.

[0366] Step 8:

[0367] The user reviews the suggestions and scripts from the server and approves or adjusts their execution as needed. If the user approves, the script is applied and repetitive tasks are automated.

[0368] Step 9:

[0369] The server collects user behavior and emotional history and automatically generates daily reports. These reports include feedback for business improvement and suggestions based on the user's emotional state.

[0370] Step 10:

[0371] Through the daily reports provided, users gain insights into their own work efficiency and emotional management. This allows them to explore even more efficient ways of working.

[0372] (Example 2)

[0373] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0374] Conventional systems were limited to automating repetitive tasks based on user input and suggesting improvements to operational efficiency, but they failed to provide feedback that took into account the user's emotional state. As a result, even if operational efficiency improved, the user's mental stress and dissatisfaction remained unresolved.

[0375] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0376] In this invention, the server includes means for collecting and storing information related to user operations, means for identifying and generating procedures for automating repetitive tasks, and means for analyzing the user's emotional state and providing feedback. This makes it possible to not only improve work efficiency but also to consider the user's emotional state, thereby reducing stress and providing a comfortable work environment.

[0377] "Information related to user operations" refers to data generated when a user uses electronic devices or applications, and includes operation history and usage time.

[0378] A "computer device" is a device equipped with hardware and software for data processing, and is primarily used as a server.

[0379] A "storage device" is a device that has the function of saving data, and includes hard disks and SSDs.

[0380] "Means for identifying repetitive tasks" refers to a function that uses data analysis techniques to find patterns and routines from user operation data and extracts repetitive tasks as tasks that can be automated.

[0381] "Means of generating procedures" refers to the ability to assemble the operations and processes necessary to streamline or automate identified repetitive tasks.

[0382] "Means for analyzing emotional states" refers to technologies that detect emotions from a user's facial expressions, voice, etc., and evaluate that state.

[0383] "Means of providing feedback" refers to a function that presents users with advice and improvement suggestions tailored to their current state and work content.

[0384] "Proposals for improving business efficiency" involve presenting more efficient and rational operating methods and improvement measures based on an analysis of the user's work and processes.

[0385] "Methods for automatically generating reports" refers to a function that automatically creates daily reports, monthly reports, etc., in a specified format based on the user's work history data.

[0386] This invention is a system that comprehensively analyzes user operation data and emotional information to achieve both improved work efficiency and emotional support for users. Specific embodiments are described below.

[0387] First, the device collects user activity data. This includes information such as smartphone and personal computer usage history and application launch times. This data is encrypted using the SSL / TLS protocol within the device and then sent to the server.

[0388] The server stores the received data in a relational database. This data is analyzed using data mining techniques to identify recurring operation patterns. Based on the results of this analysis, procedures for automating repetitive tasks are generated and presented to the user.

[0389] Simultaneously, the device uses its camera and microphone to analyze the user's emotional state in real time. The emotion engine analyzes facial expressions and tone of voice from this data to evaluate how the user is feeling about their current task.

[0390] The server provides feedback to the user based on the analysis results of the emotion engine. For example, if the user is feeling stressed about their work, it may offer relaxing music or suggest automation to reduce their workload.

[0391] For example, if a user is feeling burdened by a particular task, the server will generate an automation script for that task and present it to the user. Furthermore, as feedback that takes the user's emotional state into consideration, relaxing music may be recommended.

[0392] As an example of using a generative AI model, here is an example of a prompt: "Please suggest a script to automate a specific operation that the user finds stressful, and recommend relaxing music."

[0393] Thus, the invention comprehensively utilizes user operation data and emotional information to improve not only work efficiency but also user comfort and productivity.

[0394] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0395] Step 1:

[0396] The terminal collects user operation data. Specifically, the terminal monitors the user's application usage history and operation time in real time and records it in a database. The input is user operation information, and the output is stored in the format of date, time, application name, and usage time.

[0397] Step 2:

[0398] The device collects the user's location information. Using its built-in GPS sensor, the device determines the user's movement path and current location. Input is GPS sensor data, and output is geographical coordinate information and movement history. This data is temporarily stored within the device.

[0399] Step 3:

[0400] The device encrypts the collected data and sends it to the server. Specifically, it securely encrypts the data using the SSL / TLS protocol and sends it to the server over the internet. The input is collected operation data and location information, and the output is an encrypted data stream.

[0401] Step 4:

[0402] The server stores the received data in a database. The server verifies the incoming data in real time, confirms its integrity, and then stores it in the relational database. The input is encrypted data sent from the terminal, and the output is organized data entries in the database.

[0403] Step 5:

[0404] The server analyzes the stored data and identifies recurring operations. Here, a data analysis algorithm is used to identify frequently used operations. The input is the operation history in the database, and the output is a list of frequent patterns. This result reveals which tasks can be made more efficient.

[0405] Step 6:

[0406] The server generates and presents to the user a script to automate the identified repetitive tasks. The server uses a generation AI model to automatically generate code that automates the repetitive tasks. The input is the repetitive patterns from the analysis results, and the output is an automation script in a user-friendly format.

[0407] Step 7:

[0408] The device analyzes the user's emotional state. The device uses a camera and microphone to collect the user's facial expressions and voice, and analyzes them using an emotion engine. The input is video and audio data, and the output is metadata indicating the user's emotional state.

[0409] Step 8:

[0410] The server provides feedback based on data from the emotion engine. Based on the emotion analysis results, the server recommends music to reduce stress and suggests improvements to work processes. The input is data on the user's emotional state, and the output provides the user with recommended music and specific action plans.

[0411] (Application Example 2)

[0412] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0413] In modern work environments, there is a widespread demand for automation to improve user efficiency. However, conventional systems often automate and streamline tasks without considering the user's emotional state, which can actually increase stress. Therefore, it is necessary to improve work efficiency while simultaneously reducing the user's mental burden and providing a comfortable work environment.

[0414] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0415] In this invention, the server includes means for monitoring and collecting data related to the user's actions and emotional state; means for encrypting the collected data and transmitting it to a data management server and storing it in a data storage device; and means for analyzing the stored data and identifying repetitive tasks and emotional changes. This makes it possible to adjust the user's workload and provide encouraging information, thereby improving work efficiency and reducing mental burden.

[0416] "User actions" refer to a series of actions or instructions that a user performs on a device or system.

[0417] "Emotional state" refers to information that indicates the user's psychological state, derived from their facial expressions and voice.

[0418] A "data management server" is an external server that processes and stores information related to user actions and emotions.

[0419] A "data storage device" is a storage medium for stably and securely storing collected data.

[0420] "Saving" refers to the process of ensuring that acquired data is not lost.

[0421] "Analysis" refers to the process of using collected data to derive trends and characteristics.

[0422] "Repetitive tasks" refer to similar actions or processes that users frequently perform on the system.

[0423] "Emotional changes" refer to how a user's emotions evolve over time.

[0424] A "processing procedure" refers to a series of automated steps designed to streamline repetitive tasks.

[0425] "Information" refers to suggestions based on the user's emotional state, aimed at adjusting workload and providing encouragement.

[0426] The system for realizing this invention aims to adjust the user's workload and reduce mental burden by collecting data related to the user's actions and emotional state and performing efficient information processing. The system mainly consists of a terminal, a data management server, an emotion analysis engine, and a data storage device.

[0427] The device is responsible for capturing the user's emotional state through user interaction and the use of its camera and microphone. The emotion analysis engine uses this data to determine the user's emotions in real time and record changes in those emotions. Face detection using OpenCV and speech analysis using TensorFlow are also performed.

[0428] The data management server encrypts and securely stores the collected data. The Python Cryptography library is used, and data storage is performed by a data storage device. The server also performs detailed analysis using the stored data to identify repetitive tasks and changes in sentiment. This involves data analysis using Python libraries such as Pandas and NumPy, generating user-friendly workflows.

[0429] For example, if the emotion analysis engine determines that a user is experiencing stress during a particular task, the server uses this information to appropriately adjust the workload and generate and display an encouraging message on the terminal. The system uses a generative AI model and employs this prompt: "Based on input data from the camera and microphone, determine the worker's emotional state. If signs of fatigue or stress are detected, recommend an appropriate encouraging message and temporary task automation."

[0430] In this way, it is possible to optimize the user's work environment, increase productivity, and continue to support their mental health.

[0431] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0432] Step 1:

[0433] The device collects the user's emotional state using user actions and its camera and microphone. This utilizes sensors and input devices on the device. The input data includes the user's behavioral patterns, facial expressions, and voice tone. This input data is converted into an analyzable format through signal processing and digital conversion.

[0434] Step 2:

[0435] The terminal encrypts the collected data and sends it to the data management server in a secure state. The Python Cryptography library is used for encryption. This ensures that the user's personal information is protected when the data is sent to the server. The server receives this data and stores it in a data storage device.

[0436] Step 3:

[0437] The server analyzes the stored data using a data management system. Here, Python's Pandas and NumPy are used to analyze the data and identify patterns in repetitive tasks and emotional changes. The stored data is processed using the aforementioned tools for data analysis, and the output provides characteristic information about repetitive tasks and emotional changes.

[0438] Step 4:

[0439] The server generates information for adjusting workload and providing emotional feedback based on identified task characteristics and emotional change data. This includes a process that uses a generative AI model and prompt statements to generate appropriate responses and suggestions for task automation. This generated information is then sent back to the terminal.

[0440] Step 5:

[0441] Based on information received from the server, the terminal suggests adjusting the user's workload and displays encouraging messages. For example, it might send specific messages like "Let's take a short break" or notifications such as "An automated task will begin." As output, the suggestions are reflected on the user's screen, providing feedback to the user's actions.

[0442] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0443] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0444] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0445] [Third Embodiment]

[0446] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0447] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0448] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0449] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0450] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0451] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0452] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0453] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0454] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0455] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0456] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0457] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0458] This invention provides an embodiment of an AI system that streamlines operations and actions performed by users on a daily basis, thereby improving productivity. This system consists of a terminal, a server, and a program for coordinating them.

[0459] At system startup, the terminal monitors data related to user actions. This includes information such as keyboard and mouse usage and changes in the active window, and can be monitored in real time. Furthermore, the terminal uses the smartphone's location information to record the user's movement history. This information forms the basis for understanding the user's daily behavior.

[0460] Data collected by the device is periodically encrypted and securely transmitted to the server. The server stores this data in a database and manages it with appropriate tags. After the data is stored, advanced analytical processing is performed on the server. This analysis identifies recurring work patterns of the user and routes that can be optimized for movement.

[0461] Based on the analysis, the server generates automation scenarios and improvement suggestions to help users improve their work efficiency. This includes scripts to automate routine tasks performed on a daily basis. The server presents these suggestions and scripts to the user and notifies them through applications and email.

[0462] As a concrete example, if a user performs the task of entering specific data into a spreadsheet every morning, the terminal records this data, and the server generates a script to simplify the task. This script allows the user to complete the task with a single button click, without having to repeat the same task each time. Furthermore, the server analyzes the user's travel history and suggests the shortest route and reduces travel time, enabling them to dedicate wasted time to more creative activities.

[0463] Based on the user's daily activities, the server automatically generates daily reports, supporting the user in understanding the overall picture of streamlined operations. In this way, the present invention provides comprehensive support to reduce the user's workload and improve the creativity and efficiency of their work.

[0464] The following describes the processing flow.

[0465] Step 1:

[0466] The terminal monitors and collects data related to user actions in real time. This is done by recording information such as keyboard input, mouse clicks, and changes in window focus.

[0467] Step 2:

[0468] The device periodically acquires location information from the user's smartphone and records their movement history. This allows the system to understand the user's daily movement patterns.

[0469] Step 3:

[0470] The device packages all collected data at regular intervals and encrypts it for security purposes.

[0471] Step 4:

[0472] The terminal sends an encrypted data package to the server via the internet. A confirmation of transmission is performed to ensure the reliability of the data transfer.

[0473] Step 5:

[0474] The server decompresses the received data and saves it to the database in the appropriate format. The data is categorized and managed as application usage history, location information, and operation records.

[0475] Step 6:

[0476] The server analyzes the stored data. Specifically, it uses machine learning algorithms to detect specific patterns and repetitive tasks. The insights gained from this analysis are stored in a database.

[0477] Step 7:

[0478] Based on the analysis results, the server identifies areas where business processes can be streamlined and generates automation scripts. These scripts are intended to automate the user's own tasks.

[0479] Step 8:

[0480] The server notifies the user of generated scripts and efficiency suggestions. Notifications are sent via a dedicated application or email.

[0481] Step 9:

[0482] Users review the received proposals and scripts and incorporate them into their workflow as needed. Once the user approves, the proposed automation script is executed.

[0483] Step 10:

[0484] The server automatically generates and provides daily reports to users based on their activity history. These reports include information on work progress and suggestions for improvement.

[0485] (Example 1)

[0486] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0487] In today's technological environment, users routinely perform many repetitive tasks and movements, which require significant time and effort, posing a challenge to productivity. To streamline and automate these daily tasks, technologies are needed that accurately capture, analyze, and propose solutions for these work patterns and movements. However, current methods lack the means to handle these tasks comprehensively and safely.

[0488] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0489] In this invention, the server includes means for acquiring information related to user operations, means for transferring the acquired information to a remote storage device and storing it in a data set, and means for analyzing the stored information and recognizing repetitive tasks. This enables the user to automate daily repetitive tasks, improve travel efficiency, and save time and effort.

[0490] "Information related to user actions" refers to data about user actions on a computer or terminal, such as keyboard input, mouse clicks, and changes in the active window.

[0491] A "remote storage device" refers to a storage system accessible via a network, such as a server or a cloud-based database, for storing collected data.

[0492] A "data set" is a collection of data that has been gathered and stored, managed together, and used for later analysis and processing.

[0493] "Repetitive tasks" refer to similar operations or tasks that users perform many times on a daily basis, and identifying these tasks makes them suitable for automation.

[0494] "Processing code" refers to scripts or programs generated to automate specific repetitive tasks, serving as a means to simplify the user's work.

[0495] "Suggestions for improving travel efficiency" are specific advice to eliminate waste by analyzing the user's travel history to determine the optimal travel route and time.

[0496] This invention is a system for streamlining user operations and movement, and consists of a terminal, a server, and a program to link them together. In this system, the terminal monitors information related to user operations in real time. Specifically, it uses dedicated monitoring software to log keyboard input, mouse operations, and window activity status. In addition, a smartphone is used as a terminal, and location information services are utilized to continuously acquire the user's movement history.

[0497] Information collected by the device is encrypted to ensure data security. Standard encryption algorithms such as AES-256 are used for encryption. The encrypted information is then periodically sent to the server via the secure HTTPS protocol. The server stores the received information in a database and tags it with metadata such as date and data type.

[0498] The server analyzes stored information using a generated AI model. This analysis identifies recurring work patterns and travel routes of the user. Based on the identified work patterns, the server generates processing code to automate the user's routine tasks. The generated processing code is a means to simplify and automate the user's work. In addition, the server analyzes the user's travel history and makes suggestions for more efficient travel.

[0499] For example, if a user enters data into a spreadsheet from a specific website every morning, the terminal records this work pattern. Based on this, the server generates processing code to automate the data entry process and provides it to the user. This prompt allows the user to complete the task with a single click, improving productivity. Additionally, the system suggests optimal routes through analysis of travel history, contributing to shorter commute times and reduced unnecessary travel.

[0500] An example of a prompt message is, "Create an automated spreadsheet input scenario based on the user's work patterns over the past week." This structure allows users to enjoy increased efficiency in their daily tasks.

[0501] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0502] Step 1:

[0503] Data collection by devices

[0504] The terminal monitors user activity in real time and collects data. Specifically, it uses monitoring software to acquire keyboard input, mouse movements, and window activity status. Input is user activity events, and output is a log file containing this data. In this step, the terminal continuously records the user's daily activities.

[0505] Step 2:

[0506] Location information collection by the device

[0507] The device uses the smartphone's location information to collect the user's movement history. It utilizes GPS data and Wi-Fi location information to record the user's current location and travel route. The input is the user's location data, and the output is a collection of data points representing the movement history. This allows the device to create a foundation for understanding the user's movement patterns.

[0508] Step 3:

[0509] Data encryption and transmission

[0510] The device encrypts the collected data and sends it to the server. Encryption algorithms such as AES-256 are used to secure the data. The input is the previously collected operation and location data, and the output is an encrypted data package. This data is sent to the server via the HTTPS protocol.

[0511] Step 4:

[0512] Server-based data storage and tagging

[0513] The server receives encrypted data sent from the terminal and stores it in a database. The data is tagged with metadata such as date and type, preparing it for subsequent processing. The input is an encrypted data package, and the output is a tagged database entry. This enables data organization and efficient management.

[0514] Step 5:

[0515] Server-based data analysis

[0516] The server uses a generative AI model based on stored data to recognize the user's repetitive tasks and travel routes. The input is stored database entries, and the output is recognized work patterns and optimization suggestions. This allows the server to present the user with necessary improvement suggestions.

[0517] Step 6:

[0518] Generating automation scripts

[0519] The server generates automation scripts for identified repetitive tasks. It leverages a generative AI model to create code that streamlines user operations. The input is a recognized work pattern, and the output is an automation script. The server uses this script to simplify the user's tasks.

[0520] Step 7:

[0521] Improvement suggestions and script notifications

[0522] The server notifies the user of the generated scripts and suggestions for improving travel efficiency. The input is the generated scripts and suggestions, and the output is a notification message to the user. This allows the user to improve the efficiency of their work based on the suggestions.

[0523] (Application Example 1)

[0524] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0525] In factories, robot operation and route selection often rely on experience and intuition, making efficient operation difficult. Furthermore, repetitive tasks and unnecessary movements in daily operations can reduce productivity and lead to a lack of optimization, which needs to be addressed.

[0526] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0527] In this invention, the server includes means for monitoring and collecting data related to user operations, means for transmitting the collected data to a communication device and storing it in a storage device, means for analyzing the stored data to identify repetitive tasks, means for generating instructions to automate the identified repetitive tasks, means for analyzing the robot's operation history to identify optimized operation patterns, and means for generating instructions to optimize the operation path based on the identified operation patterns. This significantly improves the operational efficiency of robots in a factory, enabling reduced working time and increased productivity.

[0528] "Data related to user actions" refers to all information related to user actions on a computer or device, such as keyboard input, mouse operations, and changes to the active window.

[0529] "Means for transmitting collected data to a communication device and storing it in a storage device" refers to technical methods and devices for securely transmitting and storing data collected by a terminal device to a storage device such as a server via a network.

[0530] "Means for identifying repetitive tasks" refers to methods and devices for analyzing stored data to detect and identify routine tasks that users perform on a daily basis.

[0531] "Instructions for automation" refers to the content of instructions for generating specific operational procedures or scripts to automate and streamline identified tasks.

[0532] "Robot operation history" refers to a record of actions and route selections performed by a robot in the past, and is information used to improve work efficiency.

[0533] "Optimized motion patterns" refer to the optimized robot movements and route selection methods obtained as a result of analysis.

[0534] "Instructions for optimizing the movement path" refers to instructions given to a robot to select and guide it to the shortest route and a movement path that takes obstacles into consideration, in order to reach its destination efficiently.

[0535] The system for implementing this invention aims to efficiently collect and analyze user operation and activity history. The terminal monitors data related to user operations in real time and records information such as keyboard input and mouse operations. Furthermore, it utilizes location information to track the user's movement history and construct a detailed activity log.

[0536] The collected data is securely transmitted to the server via a communication device after undergoing an encryption process. The server stores the data in dedicated storage devices and then efficiently tags it in a database. The server then uses AI models and algorithms to analyze the data and automatically identify repetitive user tasks and optimize travel routes. To achieve this, the system utilizes sensors and GPS devices as hardware, and Python, AWS SageMaker, and DynamoDB as software.

[0537] Based on the analysis results, the server generates specific instructions for automation and operational efficiency. For robots, it suggests the optimal movement path based on the optimized movement patterns and issues instructions to immediately execute those patterns. Finally, these instructions are delivered via communication to the user's mobile device or robot control system, thereby improving operational efficiency.

[0538] As a concrete example, in parts picking operations in a factory, it is possible to instruct robots on the shortest route and most efficient operating procedures by analyzing historical data accumulated by the user. This can significantly reduce picking time and improve productivity.

[0539] Examples of prompts for a generative AI model:

[0540] "Based on past movement data, please provide a detailed description of the most efficient route for parts picking. The goal is to complete the picking in the shortest possible time. Please also propose alternative routes in case of obstacles."

[0541] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0542] Step 1:

[0543] The terminal monitors user actions in real time, collecting information on keyboard and mouse operations, as well as the activity of the active window. This information is recorded as a user action dataset. The input is user action information, and the output is stored as a dataset.

[0544] Step 2:

[0545] The terminal encrypts the collected data using AES encryption and securely transmits it to the server via a communication device. This process ensures data security. The input is the unencrypted operation data, and the output is the encrypted data.

[0546] Step 3:

[0547] The server decrypts the received encrypted data and stores it in a database. Appropriate tagging is applied during storage to ensure efficient use in subsequent processing. The input is encrypted data, and the output is a tagged database entry.

[0548] Step 4:

[0549] The server uses stored data to leverage an AI model and perform data analysis. This identifies repetitive user tasks and discovers patterns that can be improved. The input is database entries, and the output is the identified improvement patterns.

[0550] Step 5:

[0551] Based on this analysis, the server generates instructions that can be automated and analyzes the robot's motion history to identify more efficient motion patterns. The instructions obtained through this process are compiled as a suggestion for the robot's optimal motion path. The input is the improved pattern, and the output is the content of the instructions.

[0552] Step 6:

[0553] The server notifies the robot control system and the user's device of the generated instructions. The user and robot then perform specific actions based on these instructions, improving productivity. The input is the generated instructions, and the output is the actual operational action.

[0554] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0555] This invention describes a configuration that provides more personalized support to users by combining an emotion engine with an AI system that analyzes and optimizes user operations and behavior. The system consists of a user terminal, a server, and a program that integrates the emotion engine.

[0556] At system startup, the terminal collects user activity and location information. This data includes information about the applications the user is running and location information obtained from the smartphone. The terminal encrypts this data and sends it to the server.

[0557] The server stores the received data in a database. The stored data is then analyzed in detail to identify repetitive operations, and suggestions for improving operational efficiency are created based on the results. In particular, scripts for automating repetitive tasks are generated and presented to the user.

[0558] The emotion engine recognizes the user's emotions and provides feedback tailored to the user's state based on that information. For example, the device analyzes the user's facial expressions and tone of voice through the camera and voice input to detect changes in emotion. The server takes in the data obtained from the emotion engine and analyzes the user's emotional history. This allows for new approaches to reduce the frequency of negative emotions and suggestions for improving work processes.

[0559] For example, if a user seems to be finding a task burdensome, the server might suggest automating that task, and the emotion engine might provide feedback such as recommending relaxing music. Furthermore, the system adjusts the content of the automation script in consideration of changes in the user's emotions to help them perform their work comfortably.

[0560] Thus, by considering both user behavior and emotions, the present invention achieves comprehensive user support that goes beyond simply improving work efficiency, and helps users work in a more comfortable and productive environment.

[0561] The following describes the processing flow.

[0562] Step 1:

[0563] The device monitors user activity and collects application usage and keyboard and mouse input information. It also obtains the smartphone's location information and records the user's daily movement patterns.

[0564] Step 2:

[0565] The device uses its camera and microphone to acquire data in real time, including the user's facial expressions and voice tone, which are then input into the emotion engine. Based on this data, the system determines the user's emotional state.

[0566] Step 3:

[0567] The device integrates and packages collected operational data, location information, and emotional data, and securely transmits it to the server. During this process, the data is encrypted to ensure security.

[0568] Step 4:

[0569] The server decompresses the received data, categorizes it according to its relevant category, and stores it in the database. This data is managed as operation history, movement history, and sentiment history.

[0570] Step 5:

[0571] The server analyzes the stored operation data to identify patterns in repetitive tasks. Based on these patterns, it identifies parts that can be automated and generates scripts.

[0572] Step 6:

[0573] The server analyzes data from the emotion engine to investigate the user's emotional history. This analysis identifies tasks and times of day when the user is likely to experience stress.

[0574] Step 7:

[0575] The server generates automation scripts for identified repetitive tasks and suggestions for improving operations based on sentiment analysis, and notifies the user. These notifications include reminders and specific actions.

[0576] Step 8:

[0577] The user reviews the suggestions and scripts from the server and approves or adjusts their execution as needed. If the user approves, the script is applied and repetitive tasks are automated.

[0578] Step 9:

[0579] The server collects user behavior and emotional history and automatically generates daily reports. These reports include feedback for business improvement and suggestions based on the user's emotional state.

[0580] Step 10:

[0581] Through the daily reports provided, users gain insights into their own work efficiency and emotional management. This allows them to explore even more efficient ways of working.

[0582] (Example 2)

[0583] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0584] Conventional systems were limited to automating repetitive tasks based on user input and suggesting improvements to operational efficiency, but they failed to provide feedback that took into account the user's emotional state. As a result, even if operational efficiency improved, the user's mental stress and dissatisfaction remained unresolved.

[0585] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0586] In this invention, the server includes means for collecting and storing information related to user operations, means for identifying and generating procedures for automating repetitive tasks, and means for analyzing the user's emotional state and providing feedback. This makes it possible to not only improve work efficiency but also to consider the user's emotional state, thereby reducing stress and providing a comfortable work environment.

[0587] "Information related to user operations" refers to data generated when a user uses electronic devices or applications, and includes operation history and usage time.

[0588] A "computer device" is a device equipped with hardware and software for data processing, and is primarily used as a server.

[0589] A "storage device" is a device that has the function of saving data, and includes hard disks and SSDs.

[0590] "Means for identifying repetitive tasks" refers to a function that uses data analysis techniques to find patterns and routines from user operation data and extracts repetitive tasks as tasks that can be automated.

[0591] "Means of generating procedures" refers to the ability to assemble the operations and processes necessary to streamline or automate identified repetitive tasks.

[0592] "Means for analyzing emotional states" refers to technologies that detect emotions from a user's facial expressions, voice, etc., and evaluate that state.

[0593] "Means of providing feedback" refers to a function that presents users with advice and improvement suggestions tailored to their current state and work content.

[0594] "Proposals for improving business efficiency" involve presenting more efficient and rational operating methods and improvement measures based on an analysis of the user's work and processes.

[0595] "Methods for automatically generating reports" refers to a function that automatically creates daily reports, monthly reports, etc., in a specified format based on the user's work history data.

[0596] This invention is a system that comprehensively analyzes user operation data and emotional information to achieve both improved work efficiency and emotional support for users. Specific embodiments are described below.

[0597] First, the device collects user activity data. This includes information such as smartphone and personal computer usage history and application launch times. This data is encrypted using the SSL / TLS protocol within the device and then sent to the server.

[0598] The server stores the received data in a relational database. This data is analyzed using data mining techniques to identify recurring operation patterns. Based on the results of this analysis, procedures for automating repetitive tasks are generated and presented to the user.

[0599] Simultaneously, the device uses its camera and microphone to analyze the user's emotional state in real time. The emotion engine analyzes facial expressions and tone of voice from this data to evaluate how the user is feeling about their current task.

[0600] The server provides feedback to the user based on the analysis results of the emotion engine. For example, if the user is feeling stressed about their work, it may offer relaxing music or suggest automation to reduce their workload.

[0601] For example, if a user is feeling burdened by a particular task, the server will generate an automation script for that task and present it to the user. Furthermore, as feedback that takes the user's emotional state into consideration, relaxing music may be recommended.

[0602] As an example of using a generative AI model, here is an example of a prompt: "Please suggest a script to automate a specific operation that the user finds stressful, and recommend relaxing music."

[0603] Thus, the invention comprehensively utilizes user operation data and emotional information to improve not only work efficiency but also user comfort and productivity.

[0604] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0605] Step 1:

[0606] The terminal collects user operation data. Specifically, the terminal monitors the user's application usage history and operation time in real time and records it in a database. The input is user operation information, and the output is stored in the format of date, time, application name, and usage time.

[0607] Step 2:

[0608] The device collects the user's location information. Using its built-in GPS sensor, the device determines the user's movement path and current location. Input is GPS sensor data, and output is geographical coordinate information and movement history. This data is temporarily stored within the device.

[0609] Step 3:

[0610] The device encrypts the collected data and sends it to the server. Specifically, it securely encrypts the data using the SSL / TLS protocol and sends it to the server over the internet. The input is collected operation data and location information, and the output is an encrypted data stream.

[0611] Step 4:

[0612] The server stores the received data in a database. The server verifies the incoming data in real time, confirms its integrity, and then stores it in the relational database. The input is encrypted data sent from the terminal, and the output is organized data entries in the database.

[0613] Step 5:

[0614] The server analyzes the stored data and identifies recurring operations. Here, a data analysis algorithm is used to identify frequently used operations. The input is the operation history in the database, and the output is a list of frequent patterns. This result reveals which tasks can be made more efficient.

[0615] Step 6:

[0616] The server generates and presents to the user a script to automate the identified repetitive tasks. The server uses a generation AI model to automatically generate code that automates the repetitive tasks. The input is the repetitive patterns from the analysis results, and the output is an automation script in a user-friendly format.

[0617] Step 7:

[0618] The device analyzes the user's emotional state. The device uses a camera and microphone to collect the user's facial expressions and voice, and analyzes them using an emotion engine. The input is video and audio data, and the output is metadata indicating the user's emotional state.

[0619] Step 8:

[0620] The server provides feedback based on data from the emotion engine. Based on the emotion analysis results, the server recommends music to reduce stress and suggests improvements to work processes. The input is data on the user's emotional state, and the output provides the user with recommended music and specific action plans.

[0621] (Application Example 2)

[0622] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0623] In modern work environments, there is a widespread demand for automation to improve user efficiency. However, conventional systems often automate and streamline tasks without considering the user's emotional state, which can actually increase stress. Therefore, it is necessary to improve work efficiency while simultaneously reducing the user's mental burden and providing a comfortable work environment.

[0624] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0625] In this invention, the server includes means for monitoring and collecting data related to the user's actions and emotional state; means for encrypting the collected data and transmitting it to a data management server and storing it in a data storage device; and means for analyzing the stored data and identifying repetitive tasks and emotional changes. This makes it possible to adjust the user's workload and provide encouraging information, thereby improving work efficiency and reducing mental burden.

[0626] "User actions" refer to a series of actions or instructions that a user performs on a device or system.

[0627] "Emotional state" refers to information that indicates the user's psychological state, derived from their facial expressions and voice.

[0628] A "data management server" is an external server that processes and stores information related to user actions and emotions.

[0629] A "data storage device" is a storage medium for stably and securely storing collected data.

[0630] "Saving" refers to the process of ensuring that acquired data is not lost.

[0631] "Analysis" refers to the process of using collected data to derive trends and characteristics.

[0632] "Repetitive tasks" refer to similar actions or processes that users frequently perform on the system.

[0633] "Emotional changes" refer to how a user's emotions evolve over time.

[0634] A "processing procedure" refers to a series of automated steps designed to streamline repetitive tasks.

[0635] "Information" refers to suggestions based on the user's emotional state, aimed at adjusting workload and providing encouragement.

[0636] The system for realizing this invention aims to adjust the user's workload and reduce mental burden by collecting data related to the user's actions and emotional state and performing efficient information processing. The system mainly consists of a terminal, a data management server, an emotion analysis engine, and a data storage device.

[0637] The device is responsible for capturing the user's emotional state through user interaction and the use of its camera and microphone. The emotion analysis engine uses this data to determine the user's emotions in real time and record changes in those emotions. Face detection using OpenCV and speech analysis using TensorFlow are also performed.

[0638] The data management server encrypts and securely stores the collected data. The Python Cryptography library is used, and data storage is performed by a data storage device. The server also performs detailed analysis using the stored data to identify repetitive tasks and changes in sentiment. This involves data analysis using Python libraries such as Pandas and NumPy, generating user-friendly workflows.

[0639] For example, if the emotion analysis engine determines that a user is experiencing stress during a particular task, the server uses this information to appropriately adjust the workload and generate and display an encouraging message on the terminal. The system uses a generative AI model and employs this prompt: "Based on input data from the camera and microphone, determine the worker's emotional state. If signs of fatigue or stress are detected, recommend an appropriate encouraging message and temporary task automation."

[0640] In this way, it is possible to optimize the user's work environment, increase productivity, and continue to support their mental health.

[0641] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0642] Step 1:

[0643] The device collects the user's emotional state using user actions and its camera and microphone. This utilizes sensors and input devices on the device. The input data includes the user's behavioral patterns, facial expressions, and voice tone. This input data is converted into an analyzable format through signal processing and digital conversion.

[0644] Step 2:

[0645] The terminal encrypts the collected data and sends it to the data management server in a secure state. The Python Cryptography library is used for encryption. This ensures that the user's personal information is protected when the data is sent to the server. The server receives this data and stores it in a data storage device.

[0646] Step 3:

[0647] The server analyzes the stored data using a data management system. Here, Python's Pandas and NumPy are used to analyze the data and identify patterns in repetitive tasks and emotional changes. The stored data is processed using the aforementioned tools for data analysis, and the output provides characteristic information about repetitive tasks and emotional changes.

[0648] Step 4:

[0649] The server generates information for adjusting workload and providing emotional feedback based on identified task characteristics and emotional change data. This includes a process that uses a generative AI model and prompt statements to generate appropriate responses and suggestions for task automation. This generated information is then sent back to the terminal.

[0650] Step 5:

[0651] Based on information received from the server, the terminal suggests adjusting the user's workload and displays encouraging messages. For example, it might send specific messages like "Let's take a short break" or notifications such as "An automated task will begin." As output, the suggestions are reflected on the user's screen, providing feedback to the user's actions.

[0652] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0653] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0654] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0655] [Fourth Embodiment]

[0656] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0657] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0658] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0659] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0660] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0661] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0662] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0663] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0664] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0665] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0666] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0667] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0668] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0669] This invention provides an embodiment of an AI system that streamlines operations and actions performed by users on a daily basis, thereby improving productivity. This system consists of a terminal, a server, and a program for coordinating them.

[0670] At system startup, the terminal monitors data related to user actions. This includes information such as keyboard and mouse usage and changes in the active window, and can be monitored in real time. Furthermore, the terminal uses the smartphone's location information to record the user's movement history. This information forms the basis for understanding the user's daily behavior.

[0671] Data collected by the device is periodically encrypted and securely transmitted to the server. The server stores this data in a database and manages it with appropriate tags. After the data is stored, advanced analytical processing is performed on the server. This analysis identifies recurring work patterns of the user and routes that can be optimized for movement.

[0672] Based on the analysis, the server generates automation scenarios and improvement suggestions to help users improve their work efficiency. This includes scripts to automate routine tasks performed on a daily basis. The server presents these suggestions and scripts to the user and notifies them through applications and email.

[0673] As a concrete example, if a user performs the task of entering specific data into a spreadsheet every morning, the terminal records this data, and the server generates a script to simplify the task. This script allows the user to complete the task with a single button click, without having to repeat the same task each time. Furthermore, the server analyzes the user's travel history and suggests the shortest route and reduces travel time, enabling them to dedicate wasted time to more creative activities.

[0674] Based on the user's daily activities, the server automatically generates daily reports, supporting the user in understanding the overall picture of streamlined operations. In this way, the present invention provides comprehensive support to reduce the user's workload and improve the creativity and efficiency of their work.

[0675] The following describes the processing flow.

[0676] Step 1:

[0677] The terminal monitors and collects data related to user actions in real time. This is done by recording information such as keyboard input, mouse clicks, and changes in window focus.

[0678] Step 2:

[0679] The device periodically acquires location information from the user's smartphone and records their movement history. This allows the system to understand the user's daily movement patterns.

[0680] Step 3:

[0681] The device packages all collected data at regular intervals and encrypts it for security purposes.

[0682] Step 4:

[0683] The terminal sends an encrypted data package to the server via the internet. A confirmation of transmission is performed to ensure the reliability of the data transfer.

[0684] Step 5:

[0685] The server decompresses the received data and saves it to the database in the appropriate format. The data is categorized and managed as application usage history, location information, and operation records.

[0686] Step 6:

[0687] The server analyzes the stored data. Specifically, it uses machine learning algorithms to detect specific patterns and repetitive tasks. The insights gained from this analysis are stored in a database.

[0688] Step 7:

[0689] Based on the analysis results, the server identifies areas where business processes can be streamlined and generates automation scripts. These scripts are intended to automate the user's own tasks.

[0690] Step 8:

[0691] The server notifies the user of generated scripts and efficiency suggestions. Notifications are sent via a dedicated application or email.

[0692] Step 9:

[0693] Users review the received proposals and scripts and incorporate them into their workflow as needed. Once the user approves, the proposed automation script is executed.

[0694] Step 10:

[0695] The server automatically generates and provides daily reports to users based on their activity history. These reports include information on work progress and suggestions for improvement.

[0696] (Example 1)

[0697] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0698] In today's technological environment, users routinely perform many repetitive tasks and movements, which require significant time and effort, posing a challenge to productivity. To streamline and automate these daily tasks, technologies are needed that accurately capture, analyze, and propose solutions for these work patterns and movements. However, current methods lack the means to handle these tasks comprehensively and safely.

[0699] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0700] In this invention, the server includes means for acquiring information related to user operations, means for transferring the acquired information to a remote storage device and storing it in a data set, and means for analyzing the stored information and recognizing repetitive tasks. This enables the user to automate daily repetitive tasks, improve travel efficiency, and save time and effort.

[0701] "Information related to user actions" refers to data about user actions on a computer or terminal, such as keyboard input, mouse clicks, and changes in the active window.

[0702] A "remote storage device" refers to a storage system accessible via a network, such as a server or a cloud-based database, for storing collected data.

[0703] A "data set" is a collection of data that has been gathered and stored, managed together, and used for later analysis and processing.

[0704] "Repetitive tasks" refer to similar operations or tasks that users perform many times on a daily basis, and identifying these tasks makes them suitable for automation.

[0705] "Processing code" refers to scripts or programs generated to automate specific repetitive tasks, serving as a means to simplify the user's work.

[0706] "Suggestions for improving travel efficiency" are specific advice to eliminate waste by analyzing the user's travel history to determine the optimal travel route and time.

[0707] This invention is a system for streamlining user operations and movement, and consists of a terminal, a server, and a program to link them together. In this system, the terminal monitors information related to user operations in real time. Specifically, it uses dedicated monitoring software to log keyboard input, mouse operations, and window activity status. In addition, a smartphone is used as a terminal, and location information services are utilized to continuously acquire the user's movement history.

[0708] Information collected by the device is encrypted to ensure data security. Standard encryption algorithms such as AES-256 are used for encryption. The encrypted information is then periodically sent to the server via the secure HTTPS protocol. The server stores the received information in a database and tags it with metadata such as date and data type.

[0709] The server analyzes stored information using a generated AI model. This analysis identifies recurring work patterns and travel routes of the user. Based on the identified work patterns, the server generates processing code to automate the user's routine tasks. The generated processing code is a means to simplify and automate the user's work. In addition, the server analyzes the user's travel history and makes suggestions for more efficient travel.

[0710] For example, if a user enters data into a spreadsheet from a specific website every morning, the terminal records this work pattern. Based on this, the server generates processing code to automate the data entry process and provides it to the user. This prompt allows the user to complete the task with a single click, improving productivity. Additionally, the system suggests optimal routes through analysis of travel history, contributing to shorter commute times and reduced unnecessary travel.

[0711] An example of a prompt message is, "Create an automated spreadsheet input scenario based on the user's work patterns over the past week." This structure allows users to enjoy increased efficiency in their daily tasks.

[0712] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0713] Step 1:

[0714] Data collection by devices

[0715] The terminal monitors user activity in real time and collects data. Specifically, it uses monitoring software to acquire keyboard input, mouse movements, and window activity status. Input is user activity events, and output is a log file containing this data. In this step, the terminal continuously records the user's daily activities.

[0716] Step 2:

[0717] Location information collection by the device

[0718] The device uses the smartphone's location information to collect the user's movement history. It utilizes GPS data and Wi-Fi location information to record the user's current location and travel route. The input is the user's location data, and the output is a collection of data points representing the movement history. This allows the device to create a foundation for understanding the user's movement patterns.

[0719] Step 3:

[0720] Data encryption and transmission

[0721] The device encrypts the collected data and sends it to the server. Encryption algorithms such as AES-256 are used to secure the data. The input is the previously collected operation and location data, and the output is an encrypted data package. This data is sent to the server via the HTTPS protocol.

[0722] Step 4:

[0723] Server-based data storage and tagging

[0724] The server receives encrypted data sent from the terminal and stores it in a database. The data is tagged with metadata such as date and type, preparing it for subsequent processing. The input is an encrypted data package, and the output is a tagged database entry. This enables data organization and efficient management.

[0725] Step 5:

[0726] Server-based data analysis

[0727] The server uses a generative AI model based on stored data to recognize the user's repetitive tasks and travel routes. The input is stored database entries, and the output is recognized work patterns and optimization suggestions. This allows the server to present the user with necessary improvement suggestions.

[0728] Step 6:

[0729] Generating automation scripts

[0730] The server generates automation scripts for identified repetitive tasks. It leverages a generative AI model to create code that streamlines user operations. The input is a recognized work pattern, and the output is an automation script. The server uses this script to simplify the user's tasks.

[0731] Step 7:

[0732] Improvement suggestions and script notifications

[0733] The server notifies the user of the generated scripts and suggestions for improving travel efficiency. The input is the generated scripts and suggestions, and the output is a notification message to the user. This allows the user to improve the efficiency of their work based on the suggestions.

[0734] (Application Example 1)

[0735] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0736] In factories, robot operation and route selection often rely on experience and intuition, making efficient operation difficult. Furthermore, repetitive tasks and unnecessary movements in daily operations can reduce productivity and lead to a lack of optimization, which needs to be addressed.

[0737] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0738] In this invention, the server includes means for monitoring and collecting data related to user operations, means for transmitting the collected data to a communication device and storing it in a storage device, means for analyzing the stored data to identify repetitive tasks, means for generating instructions to automate the identified repetitive tasks, means for analyzing the robot's operation history to identify optimized operation patterns, and means for generating instructions to optimize the operation path based on the identified operation patterns. This significantly improves the operational efficiency of robots in a factory, enabling reduced working time and increased productivity.

[0739] "Data related to user actions" refers to all information related to user actions on a computer or device, such as keyboard input, mouse operations, and changes to the active window.

[0740] "Means for transmitting collected data to a communication device and storing it in a storage device" refers to technical methods and devices for securely transmitting and storing data collected by a terminal device to a storage device such as a server via a network.

[0741] "Means for identifying repetitive tasks" refers to methods and devices for analyzing stored data to detect and identify routine tasks that users perform on a daily basis.

[0742] "Instructions for automation" refers to the content of instructions for generating specific operational procedures or scripts to automate and streamline identified tasks.

[0743] "Robot operation history" refers to a record of actions and route selections performed by a robot in the past, and is information used to improve work efficiency.

[0744] "Optimized motion patterns" refer to the optimized robot movements and route selection methods obtained as a result of analysis.

[0745] "Instructions for optimizing the movement path" refers to instructions given to a robot to select and guide it to the shortest route and a movement path that takes obstacles into consideration, in order to reach its destination efficiently.

[0746] The system for implementing this invention aims to efficiently collect and analyze user operation and activity history. The terminal monitors data related to user operations in real time and records information such as keyboard input and mouse operations. Furthermore, it utilizes location information to track the user's movement history and construct a detailed activity log.

[0747] The collected data is securely transmitted to the server via a communication device after undergoing an encryption process. The server stores the data in dedicated storage devices and then efficiently tags it in a database. The server then uses AI models and algorithms to analyze the data and automatically identify repetitive user tasks and optimize travel routes. To achieve this, the system utilizes sensors and GPS devices as hardware, and Python, AWS SageMaker, and DynamoDB as software.

[0748] Based on the analysis results, the server generates specific instructions for automation and operational efficiency. For robots, it suggests the optimal movement path based on the optimized movement patterns and issues instructions to immediately execute those patterns. Finally, these instructions are delivered via communication to the user's mobile device or robot control system, thereby improving operational efficiency.

[0749] As a concrete example, in parts picking operations in a factory, it is possible to instruct robots on the shortest route and most efficient operating procedures by analyzing historical data accumulated by the user. This can significantly reduce picking time and improve productivity.

[0750] Examples of prompts for a generative AI model:

[0751] "Based on past movement data, please provide a detailed description of the most efficient route for parts picking. The goal is to complete the picking in the shortest possible time. Please also propose alternative routes in case of obstacles."

[0752] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0753] Step 1:

[0754] The terminal monitors user actions in real time, collecting information on keyboard and mouse operations, as well as the activity of the active window. This information is recorded as a user action dataset. The input is user action information, and the output is stored as a dataset.

[0755] Step 2:

[0756] The terminal encrypts the collected data using AES encryption and securely transmits it to the server via a communication device. This process ensures data security. The input is the unencrypted operation data, and the output is the encrypted data.

[0757] Step 3:

[0758] The server decrypts the received encrypted data and stores it in a database. Appropriate tagging is applied during storage to ensure efficient use in subsequent processing. The input is encrypted data, and the output is a tagged database entry.

[0759] Step 4:

[0760] The server uses stored data to leverage an AI model and perform data analysis. This identifies repetitive user tasks and discovers patterns that can be improved. The input is database entries, and the output is the identified improvement patterns.

[0761] Step 5:

[0762] Based on this analysis, the server generates instructions that can be automated and analyzes the robot's motion history to identify more efficient motion patterns. The instructions obtained through this process are compiled as a suggestion for the robot's optimal motion path. The input is the improved pattern, and the output is the content of the instructions.

[0763] Step 6:

[0764] The server notifies the robot control system and the user's device of the generated instructions. The user and robot then perform specific actions based on these instructions, improving productivity. The input is the generated instructions, and the output is the actual operational action.

[0765] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0766] This invention describes a configuration that provides more personalized support to users by combining an emotion engine with an AI system that analyzes and optimizes user operations and behavior. The system consists of a user terminal, a server, and a program that integrates the emotion engine.

[0767] At system startup, the terminal collects user activity and location information. This data includes information about the applications the user is running and location information obtained from the smartphone. The terminal encrypts this data and sends it to the server.

[0768] The server stores the received data in a database. The stored data is then analyzed in detail to identify repetitive operations, and suggestions for improving operational efficiency are created based on the results. In particular, scripts for automating repetitive tasks are generated and presented to the user.

[0769] The emotion engine recognizes the user's emotions and provides feedback tailored to the user's state based on that information. For example, the device analyzes the user's facial expressions and tone of voice through the camera and voice input to detect changes in emotion. The server takes in the data obtained from the emotion engine and analyzes the user's emotional history. This allows for new approaches to reduce the frequency of negative emotions and suggestions for improving work processes.

[0770] For example, if a user seems to be finding a task burdensome, the server might suggest automating that task, and the emotion engine might provide feedback such as recommending relaxing music. Furthermore, the system adjusts the content of the automation script in consideration of changes in the user's emotions to help them perform their work comfortably.

[0771] Thus, by considering both user behavior and emotions, the present invention achieves comprehensive user support that goes beyond simply improving work efficiency, and helps users work in a more comfortable and productive environment.

[0772] The following describes the processing flow.

[0773] Step 1:

[0774] The device monitors user activity and collects application usage and keyboard and mouse input information. It also obtains the smartphone's location information and records the user's daily movement patterns.

[0775] Step 2:

[0776] The device uses its camera and microphone to acquire data in real time, including the user's facial expressions and voice tone, which are then input into the emotion engine. Based on this data, the system determines the user's emotional state.

[0777] Step 3:

[0778] The device integrates and packages collected operational data, location information, and emotional data, and securely transmits it to the server. During this process, the data is encrypted to ensure security.

[0779] Step 4:

[0780] The server decompresses the received data, categorizes it according to its relevant category, and stores it in the database. This data is managed as operation history, movement history, and sentiment history.

[0781] Step 5:

[0782] The server analyzes the stored operation data to identify patterns in repetitive tasks. Based on these patterns, it identifies parts that can be automated and generates scripts.

[0783] Step 6:

[0784] The server analyzes data from the emotion engine to investigate the user's emotional history. This analysis identifies tasks and times of day when the user is likely to experience stress.

[0785] Step 7:

[0786] The server generates automation scripts for identified repetitive tasks and suggestions for improving operations based on sentiment analysis, and notifies the user. These notifications include reminders and specific actions.

[0787] Step 8:

[0788] The user reviews the suggestions and scripts from the server and approves or adjusts their execution as needed. If the user approves, the script is applied and repetitive tasks are automated.

[0789] Step 9:

[0790] The server collects user behavior and emotional history and automatically generates daily reports. These reports include feedback for business improvement and suggestions based on the user's emotional state.

[0791] Step 10:

[0792] Through the daily reports provided, users gain insights into their own work efficiency and emotional management. This allows them to explore even more efficient ways of working.

[0793] (Example 2)

[0794] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0795] Conventional systems were limited to automating repetitive tasks based on user input and suggesting improvements to operational efficiency, but they failed to provide feedback that took into account the user's emotional state. As a result, even if operational efficiency improved, the user's mental stress and dissatisfaction remained unresolved.

[0796] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0797] In this invention, the server includes means for collecting and storing information related to user operations, means for identifying and generating procedures for automating repetitive tasks, and means for analyzing the user's emotional state and providing feedback. This makes it possible to not only improve work efficiency but also to consider the user's emotional state, thereby reducing stress and providing a comfortable work environment.

[0798] "Information related to user operations" refers to data generated when a user uses electronic devices or applications, and includes operation history and usage time.

[0799] A "computer device" is a device equipped with hardware and software for data processing, and is primarily used as a server.

[0800] A "storage device" is a device that has the function of saving data, and includes hard disks and SSDs.

[0801] "Means for identifying repetitive tasks" refers to a function that uses data analysis techniques to find patterns and routines from user operation data and extracts repetitive tasks as tasks that can be automated.

[0802] "Means of generating procedures" refers to the ability to assemble the operations and processes necessary to streamline or automate identified repetitive tasks.

[0803] "Means for analyzing emotional states" refers to technologies that detect emotions from a user's facial expressions, voice, etc., and evaluate that state.

[0804] "Means of providing feedback" refers to a function that presents users with advice and improvement suggestions tailored to their current state and work content.

[0805] "Proposals for improving business efficiency" involve presenting more efficient and rational operating methods and improvement measures based on an analysis of the user's work and processes.

[0806] "Methods for automatically generating reports" refers to a function that automatically creates daily reports, monthly reports, etc., in a specified format based on the user's work history data.

[0807] This invention is a system that comprehensively analyzes user operation data and emotional information to achieve both improved work efficiency and emotional support for users. Specific embodiments are described below.

[0808] First, the device collects user activity data. This includes information such as smartphone and personal computer usage history and application launch times. This data is encrypted using the SSL / TLS protocol within the device and then sent to the server.

[0809] The server stores the received data in a relational database. This data is analyzed using data mining techniques to identify recurring operation patterns. Based on the results of this analysis, procedures for automating repetitive tasks are generated and presented to the user.

[0810] Simultaneously, the device uses its camera and microphone to analyze the user's emotional state in real time. The emotion engine analyzes facial expressions and tone of voice from this data to evaluate how the user is feeling about their current task.

[0811] The server provides feedback to the user based on the analysis results of the emotion engine. For example, if the user is feeling stressed about their work, it may offer relaxing music or suggest automation to reduce their workload.

[0812] For example, if a user is feeling burdened by a particular task, the server will generate an automation script for that task and present it to the user. Furthermore, as feedback that takes the user's emotional state into consideration, relaxing music may be recommended.

[0813] As an example of using a generative AI model, here is an example of a prompt: "Please suggest a script to automate a specific operation that the user finds stressful, and recommend relaxing music."

[0814] Thus, the invention comprehensively utilizes user operation data and emotional information to improve not only work efficiency but also user comfort and productivity.

[0815] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0816] Step 1:

[0817] The terminal collects user operation data. Specifically, the terminal monitors the user's application usage history and operation time in real time and records it in a database. The input is user operation information, and the output is stored in the format of date, time, application name, and usage time.

[0818] Step 2:

[0819] The device collects the user's location information. Using its built-in GPS sensor, the device determines the user's movement path and current location. Input is GPS sensor data, and output is geographical coordinate information and movement history. This data is temporarily stored within the device.

[0820] Step 3:

[0821] The device encrypts the collected data and sends it to the server. Specifically, it securely encrypts the data using the SSL / TLS protocol and sends it to the server over the internet. The input is collected operation data and location information, and the output is an encrypted data stream.

[0822] Step 4:

[0823] The server stores the received data in a database. The server verifies the incoming data in real time, confirms its integrity, and then stores it in the relational database. The input is encrypted data sent from the terminal, and the output is organized data entries in the database.

[0824] Step 5:

[0825] The server analyzes the stored data and identifies recurring operations. Here, a data analysis algorithm is used to identify frequently used operations. The input is the operation history in the database, and the output is a list of frequent patterns. This result reveals which tasks can be made more efficient.

[0826] Step 6:

[0827] The server generates and presents to the user a script to automate the identified repetitive tasks. The server uses a generation AI model to automatically generate code that automates the repetitive tasks. The input is the repetitive patterns from the analysis results, and the output is an automation script in a user-friendly format.

[0828] Step 7:

[0829] The device analyzes the user's emotional state. The device uses a camera and microphone to collect the user's facial expressions and voice, and analyzes them using an emotion engine. The input is video and audio data, and the output is metadata indicating the user's emotional state.

[0830] Step 8:

[0831] The server provides feedback based on data from the emotion engine. Based on the emotion analysis results, the server recommends music to reduce stress and suggests improvements to work processes. The input is data on the user's emotional state, and the output provides the user with recommended music and specific action plans.

[0832] (Application Example 2)

[0833] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0834] In modern work environments, there is a widespread demand for automation to improve user efficiency. However, conventional systems often automate and streamline tasks without considering the user's emotional state, which can actually increase stress. Therefore, it is necessary to improve work efficiency while simultaneously reducing the user's mental burden and providing a comfortable work environment.

[0835] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0836] In this invention, the server includes means for monitoring and collecting data related to the user's actions and emotional state; means for encrypting the collected data and transmitting it to a data management server and storing it in a data storage device; and means for analyzing the stored data and identifying repetitive tasks and emotional changes. This makes it possible to adjust the user's workload and provide encouraging information, thereby improving work efficiency and reducing mental burden.

[0837] "User actions" refer to a series of actions or instructions that a user performs on a device or system.

[0838] "Emotional state" refers to information that indicates the user's psychological state, derived from their facial expressions and voice.

[0839] A "data management server" is an external server that processes and stores information related to user actions and emotions.

[0840] A "data storage device" is a storage medium for stably and securely storing collected data.

[0841] "Saving" refers to the process of ensuring that acquired data is not lost.

[0842] "Analysis" refers to the process of using collected data to derive trends and characteristics.

[0843] "Repetitive tasks" refer to similar actions or processes that users frequently perform on the system.

[0844] "Emotional changes" refer to how a user's emotions evolve over time.

[0845] A "processing procedure" refers to a series of automated steps designed to streamline repetitive tasks.

[0846] "Information" refers to suggestions based on the user's emotional state, aimed at adjusting workload and providing encouragement.

[0847] The system for realizing this invention aims to adjust the user's workload and reduce mental burden by collecting data related to the user's actions and emotional state and performing efficient information processing. The system mainly consists of a terminal, a data management server, an emotion analysis engine, and a data storage device.

[0848] The device is responsible for capturing the user's emotional state through user interaction and the use of its camera and microphone. The emotion analysis engine uses this data to determine the user's emotions in real time and record changes in those emotions. Face detection using OpenCV and speech analysis using TensorFlow are also performed.

[0849] The data management server encrypts and securely stores the collected data. The Python Cryptography library is used, and data storage is performed by a data storage device. The server also performs detailed analysis using the stored data to identify repetitive tasks and changes in sentiment. This involves data analysis using Python libraries such as Pandas and NumPy, generating user-friendly workflows.

[0850] For example, if the emotion analysis engine determines that a user is experiencing stress during a particular task, the server uses this information to appropriately adjust the workload and generate and display an encouraging message on the terminal. The system uses a generative AI model and employs this prompt: "Based on input data from the camera and microphone, determine the worker's emotional state. If signs of fatigue or stress are detected, recommend an appropriate encouraging message and temporary task automation."

[0851] In this way, it is possible to optimize the user's work environment, increase productivity, and continue to support their mental health.

[0852] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0853] Step 1:

[0854] The device collects the user's emotional state using user actions and its camera and microphone. This utilizes sensors and input devices on the device. The input data includes the user's behavioral patterns, facial expressions, and voice tone. This input data is converted into an analyzable format through signal processing and digital conversion.

[0855] Step 2:

[0856] The terminal encrypts the collected data and sends it to the data management server in a secure state. The Python Cryptography library is used for encryption. This ensures that the user's personal information is protected when the data is sent to the server. The server receives this data and stores it in a data storage device.

[0857] Step 3:

[0858] The server analyzes the stored data using a data management system. Here, Python's Pandas and NumPy are used to analyze the data and identify patterns in repetitive tasks and emotional changes. The stored data is processed using the aforementioned tools for data analysis, and the output provides characteristic information about repetitive tasks and emotional changes.

[0859] Step 4:

[0860] The server generates information for adjusting workload and providing emotional feedback based on identified task characteristics and emotional change data. This includes a process that uses a generative AI model and prompt statements to generate appropriate responses and suggestions for task automation. This generated information is then sent back to the terminal.

[0861] Step 5:

[0862] Based on information received from the server, the terminal suggests adjusting the user's workload and displays encouraging messages. For example, it might send specific messages like "Let's take a short break" or notifications such as "An automated task will begin." As output, the suggestions are reflected on the user's screen, providing feedback to the user's actions.

[0863] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0864] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0865] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0866] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0867] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0868] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0869] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0870] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0871] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0872] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0873] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0874] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0875] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0876] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0877] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0878] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0879] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0880] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0881] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0882] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0883] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0884] The following is further disclosed regarding the embodiments described above.

[0885] (Claim 1)

[0886] Means for monitoring and collecting data related to user actions,

[0887] A means for transmitting the collected data to a server and storing it in a database,

[0888] Means for analyzing the stored data and identifying repetitive tasks,

[0889] A means of generating scripts to automate identified repetitive tasks,

[0890] A means for presenting the aforementioned script to the user,

[0891] A system that includes this.

[0892] (Claim 2)

[0893] A means of collecting user location information and recording movement history,

[0894] A means of analyzing the aforementioned movement history and making suggestions for improving operational efficiency,

[0895] The system according to claim 1, comprising:

[0896] (Claim 3)

[0897] A method for automatically generating daily reports based on the user's work history,

[0898] A means for providing the generated daily report to the user,

[0899] The system according to claim 1, comprising:

[0900] "Example 1"

[0901] (Claim 1)

[0902] Means for obtaining information related to user actions,

[0903] Means for transferring the acquired information to a remote storage device and storing it in a data set,

[0904] Means for analyzing the stored information and recognizing repetitive tasks,

[0905] A means of creating processing code to automate recognized repetitive tasks,

[0906] Means for notifying the user of the aforementioned processing code,

[0907] A means for encrypting the stored information,

[0908] Means for collecting information about the user's movements,

[0909] A means of analyzing the user's travel history and making suggestions to improve travel efficiency,

[0910] A system that includes this.

[0911] (Claim 2)

[0912] The system according to claim 1, which provides the user with the generated processing code and suggestions for improving travel efficiency.

[0913] (Claim 3)

[0914] A means of automatically generating reports based on the user's activity history,

[0915] Means for providing the generated report to the user,

[0916] The system according to claim 1, comprising:

[0917] "Application Example 1"

[0918] (Claim 1)

[0919] Means for monitoring and collecting data related to user actions,

[0920] means for transmitting the collected data to a communication device and storing it in a storage device,

[0921] Means for analyzing the stored data and identifying repetitive tasks,

[0922] A means for generating instructions to automate identified repetitive tasks,

[0923] A means for presenting the aforementioned instructions to the user,

[0924] A means for analyzing the operation history of a robot and identifying optimized operation patterns,

[0925] Means for generating instructions to optimize the operation path based on identified operation patterns,

[0926] Means for causing the robot to execute the aforementioned instructions,

[0927] A system that includes this.

[0928] (Claim 2)

[0929] A means of collecting user location information and recording movement history,

[0930] A means for analyzing the aforementioned movement history and making suggestions for improving work efficiency,

[0931] The system according to claim 1, comprising:

[0932] (Claim 3)

[0933] A method for automatically generating reports based on the user's work history,

[0934] Means for providing the generated report to the user,

[0935] A means of presenting an optimized movement path to the user based on the robot's movement history,

[0936] The system according to claim 1, comprising:

[0937] "Example 2 of combining an emotion engine"

[0938] (Claim 1)

[0939] Means for monitoring and collecting information related to user actions,

[0940] Means for transmitting the collected information to a computer and storing it in a memory device,

[0941] Means for analyzing the stored information and identifying repetitive tasks,

[0942] A means for generating procedures to automate identified repetitive tasks,

[0943] A means of analyzing the user's emotional state,

[0944] A means of providing feedback based on user sentiment information,

[0945] A means for presenting the above procedure to the user,

[0946] A system that includes this.

[0947] (Claim 2)

[0948] A means of collecting user location information and recording movement history,

[0949] A means for analyzing the aforementioned movement history and emotional information and making suggestions for improving work efficiency,

[0950] The system according to claim 1, comprising:

[0951] (Claim 3)

[0952] A means of automatically generating reports based on the user's work history and sentiment information,

[0953] Means for providing the generated report to the user,

[0954] The system according to claim 1, comprising:

[0955] "Application example 2 when combining with an emotional engine"

[0956] (Claim 1)

[0957] Means for monitoring and collecting data related to user actions and emotional states,

[0958] A means for encrypting the collected data and transmitting it to a data management server and storing it in a data storage device,

[0959] Means for analyzing the stored data and identifying repetitive tasks and emotional changes,

[0960] A means for generating processing procedures to automate identified repetitive tasks,

[0961] A means of providing information to adjust the workload and offer encouragement according to the user's emotional state,

[0962] Means for presenting the script and information to the user,

[0963] A system that includes this.

[0964] (Claim 2)

[0965] A means for collecting the user's spatial location information and recording their movement history,

[0966] A means of analyzing the aforementioned movement history and emotional history and making suggestions for improving work efficiency and providing emotional support,

[0967] The system according to claim 1, comprising:

[0968] (Claim 3)

[0969] A means of automatically generating business reports based on the user's work history and emotional history,

[0970] A means for providing the generated business report to the user,

[0971] The system according to claim 1, comprising: [Explanation of symbols]

[0972] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means for monitoring and collecting data related to user actions, A means for transmitting the collected data to a server and storing it in a database, Means for analyzing the stored data and identifying repetitive tasks, A means of generating scripts to automate identified repetitive tasks, A means for presenting the aforementioned script to the user, A system that includes this.

2. A means of collecting user location information and recording movement history, A means of analyzing the aforementioned movement history and making suggestions for improving operational efficiency, The system according to claim 1, comprising:

3. A method for automatically generating daily reports based on the user's work history, A means for providing the generated daily report to the user, The system according to claim 1, comprising:

Images