System

Abstract

A system is provided.SOLUTION: This system includes a means for acquiring schedule data of a user, a means for specifying necessary materials on the basis of the acquired schedule data, and a means for automatically displaying the specified materials on a desktop on a terminal of the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's work environment, desktops are overflowing with a wide variety of documents and folders, making it difficult to quickly find the documents you need. This causes users to spend a lot of time searching for documents, resulting in reduced work efficiency. They also often deviate from their assigned tasks and perform unrelated tasks, further wasting time and reducing efficiency. Furthermore, there is a risk of missing important schedules and deadlines, which can cause work delays and problems. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides a system that includes a means for acquiring a user's schedule data, a means for identifying necessary materials based on the acquired schedule data, and a means for automatically displaying the identified materials on the desktop of the user's terminal. The system also includes a means for monitoring the user's activity while working and a means for displaying a warning if the monitored activity differs from the schedule data, thereby preventing the user from deviating from the scheduled work. Furthermore, the system provides a means for analyzing the user's schedule data, generating reminders as needed, and displaying them on the user's terminal, preventing the user from missing important appointments or deadlines. These means allow the user to perform their work efficiently and with concentration.

[0006] "User" refers to a person who uses a computer system or device.

[0007] "Schedule data" refers to data containing information about a user's schedule or agenda.

[0008] "Materials" refers to digital information such as documents, charts, images, and presentation files that are related to the user's business or work.

[0009] "Terminal" refers to a computer or other device that is directly operated by a user.

[0010] A "desktop" is the part of the screen interface on a user's device where icons and shortcuts are located.

[0011] "Means of monitoring" refers to technology that uses programs and algorithms to monitor user behavior and operations in real time.

[0012] "Means for displaying a warning" refers to technology that displays a pop-up or alert message to notify the user that they are in a situation that requires their attention.

[0013] "Reminder" refers to a message or notification that notifies a user about important events or deadlines.

[0014] "Means of acquisition" refers to the technology used to collect necessary information from external data sources and store it in an internal database.

[0015] "Means of identification" refers to the algorithms or processes used to select the necessary information or items based on the acquired data.

[0016] "Display means" refers to a technique for visually displaying the identified materials or information on the user's terminal screen.

[0017] "Means of analysis" refers to the technology of analyzing acquired data using statistical or machine learning techniques to extract useful information. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention is a desktop magician system for improving the user's work efficiency. It aims to acquire the user's schedule data and automatically display the necessary documents on the desktop based on that data. The system is realized by linking a server and a terminal.

[0040] System configuration

[0041] The basic system configuration is as follows:

[0042] 1. Server:

[0043] Retrieve user schedule data from external services (e.g., Google Calendar or Outlook Calendar).

[0044] The acquired schedule data is managed and saved, and the necessary data is sent to the user terminal.

[0045] Analyze schedule data and generate reminders.

[0046] 2. Terminal:

[0047] Schedule data provided by the server is received and stored in local storage.

[0048] Display the necessary materials on your desktop based on the current time.

[0049] Monitor users' work activity and display alerts when work is outside of schedule.

[0050] Notify users of reminders.

[0051] Overview of program processing

[0052] 1. Acquiring and saving schedule data

[0053] Server: When a user logs in, the server uses an API to retrieve the user's schedule data and saves it in a database. This data includes the date, time, title, and details of the event.

[0054] 2. Schedule data synchronization

[0055] Server: Updates the schedule data at regular intervals to keep it up to date.

[0056] Terminal: Receives schedule data updates from the server at regular intervals and saves them in local storage.

[0057] 3. Automatic display of materials

[0058] Terminal: Based on the current time, schedule data and related materials are identified and displayed on the desktop. For example, if a meeting is scheduled for 10:00 AM, materials related to that meeting will be displayed on the desktop 15 minutes before the meeting.

[0059] 4. Activity Monitoring and Alerts

[0060] Terminal: Monitors user activity and displays a warning pop-up after a certain period of time if the user is performing tasks that differ from the scheduled data. This pop-up prompts the user to recheck their schedule.

[0061] 5. Reminders

[0062] Server: Analyzes schedule data and generates reminders when important events or deadlines are approaching. Reminders are generated at set times (e.g., 3 days before, 1 day before).

[0063] Device: Reminder notifications are received and displayed on the desktop. After the user confirms the reminder, they can also use the snooze function to be notified again.

[0064] Specific examples

[0065] User's morning activities:

[0066] 1. A user starts up their PC in the morning and logs into Desktop Magician.

[0067] 2. Server: Obtains the user's latest schedule data from an external calendar service and sends it to the device.

[0068] 3. Terminal: Based on the received data, materials related to the meeting at 10:00 AM today are displayed on the desktop.

[0069] Monitoring during work:

[0070] 1. In the morning, the user is browsing the web for an activity other than the scheduled "report writing."

[0071] 2. Terminal: Monitor the title bar of the active window and display a warning popup if it is different from the schedule.

[0072] 3. A pop-up will appear saying, "You are currently working on something different from your schedule. Would you like to return to your schedule?" and the user will select "Yes."

[0073] Reminder:

[0074] 1. In the evening, a reminder appears on your device to let you know that a project is due tomorrow.

[0075] 2. Server: Generates reminders the day before and sends them to the device.

[0076] 3. Device: Reminders pop up on the desktop so users can take action.

[0077] As described above, the present invention provides an environment in which a user can work efficiently through a specific program and its processing.

[0078] The processing flow will be explained below.

[0079] Step 1:

[0080] The user logs in to Desktop Magician. Once logged in, the system sends the user authentication information to the server.

[0081] Step 2:

[0082] The server receives the user's authentication information and verifies that the user is a valid user. If authentication is successful, the server retrieves the user's schedule data (e.g., Google Calendar or Outlook Calendar) from an external API.

[0083] Step 3:

[0084] The server analyzes the acquired schedule data, extracts the user's schedule items, and stores them in a database, including the schedule date and time, title, details, and links to attachments.

[0085] Step 4:

[0086] The device requests synchronization of schedule data from the server at regular intervals (for example, every morning), and the server sends the latest schedule data to the device.

[0087] Step 5:

[0088] The device stores the received schedule data in local storage, then compares the current time with the schedule data to identify events scheduled within the next few hours.

[0089] Step 6:

[0090] The device will search for materials related to the identified event and automatically display them on the desktop. For example, if a meeting is scheduled for 10:00 a.m., materials related to this meeting will be displayed on the desktop 15 minutes before the meeting.

[0091] Step 7:

[0092] The device monitors the user's work activity and displays a warning popup after a specified time (e.g., 5 minutes) if the user is not performing the scheduled task.

[0093] Step 8:

[0094] The device will display a pop-up warning to the user, saying, "You are currently working on something different from your scheduled schedule. Would you like to return to your scheduled schedule?" If the user selects "Yes," they will be provided with a link to easily return to their scheduled materials or applications.

[0095] Step 9:

[0096] The server periodically analyzes schedule data and generates reminders when important events or deadlines are approaching, which are then sent to the device.

[0097] Step 10:

[0098] The device will display the received reminder as a pop-up on the desktop, and after the user confirms the reminder, they can set the snooze function to be notified again later.

[0099] This series of processes allows users to concentrate on their work efficiently and makes schedule management easier.

[0100] Example 1

[0101] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0102] Conventional schedule management systems require users to prepare materials themselves, and do not provide sufficient warnings or reminders when they are working outside of the schedule, which can lead to reduced work efficiency.

[0103] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0104] In this invention, the server includes means for acquiring user schedule data from an external service, means for saving the acquired schedule data in a database, and means for periodically updating the schedule data to keep it up to date. This allows necessary materials to be automatically displayed on the desktop based on the latest schedule data, improving the user's work efficiency.

[0105] The terminal also includes means for identifying materials relevant to the current time based on the schedule data and automatically displaying them on the desktop, means for monitoring the user's activity during work and displaying a warning if the user is performing work that differs from the schedule, means for analyzing the schedule data and generating a reminder when an important appointment or deadline approaches, and means for displaying the generated reminder on the desktop. This enables the user to accurately perform the scheduled work and proceed with work without missing important appointments or deadlines.

[0106] "User schedule data" refers to information about plans and events that a user has registered in an external service (for example, a calendar service), and includes information such as date and time, title, and details.

[0107] "External service" refers to an online platform for managing, storing, and providing users' schedule data, such as a calendar application or a cloud-based schedule management service.

[0108] A "database" is a system that organizes, stores, and manages acquired schedule data.

[0109] "Regularly updated" means that schedule data is reacquired from an external service at regular intervals to keep the contents of the database up to date.

[0110] "Necessary materials" refers to documents and files related to scheduled tasks and events based on the user's schedule data.

[0111] "Automatically displaying on the desktop" means displaying materials on the screen of the user's terminal without the need for manual operation.

[0112] "Terminal" refers to an electronic computer that allows a user to use the Desktop Magician System, and includes desktop PCs, notebook PCs, tablets, etc.

[0113] "Activity" refers to the work or operation that a user performs on a terminal, and indicates the type of work that can be determined, for example, by the title bar of an active window.

[0114] A "warning popup" is a small window that displays a warning or instructions when the user is performing work that is different from the schedule.

[0115] A "reminder" is a message that notifies the user when an important event or deadline is approaching, and is generated at a specified timing.

[0116] "Local storage" refers to a storage device (such as a hard disk or SSD) built into the user's terminal, and is used to store schedule data and related materials received from the server.

[0117] This invention is a desktop magician system for improving the user's work efficiency. The system acquires the user's schedule data and automatically displays the necessary documents on the desktop based on that data. The system is realized by linking a server and a terminal.

[0118] server

[0119] When a user logs in to the Desktop Magician system, the server uses an API to retrieve the user's schedule data from external services (such as Google Calendar or Outlook Calendar). This retrieved data includes the date, time, title, and details of the appointment. The retrieved data is then stored in a database on the server. The server then updates the schedule data at regular intervals to keep it up to date. In addition, the server analyzes the schedule data and generates reminders when important appointments or deadlines are approaching, which are then sent to the device.

[0120] Terminal

[0121] The device periodically receives schedule data provided by the server and saves it in local storage. Based on the current time, it identifies the schedule data and related materials and automatically displays them on the desktop. The device also monitors the user's work activity and displays a warning pop-up after a certain period of time if the user is performing work that differs from the schedule data. This pop-up prompts the user to reconfirm their schedule. It also sends reminder notifications, providing information for the user to take action. Reminder notifications are displayed as a pop-up on the desktop, and after the user confirms the reminder, they can use the snooze function to be notified again.

[0122] Usage example

[0123] User's morning activities:

[0124] A user starts up their PC in the morning and logs into Desktop Magician.

[0125] The server obtains the user's latest schedule data from an external calendar service and sends it to the terminal.

[0126] Based on the data received by the device, materials related to the meeting at 10:00 a.m. today will be displayed on the desktop.

[0127] Monitoring during work:

[0128] In the morning, the user is browsing the web, which is different from the scheduled "report writing" activity.

[0129] The terminal monitors the title bar of the active window and displays a warning popup if it is different from what is expected.

[0130] The popup displays "You are currently working on something different from your schedule. Would you like to return to your schedule?" and the user selects "Yes."

[0131] Reminder:

[0132] In the evening, a reminder appears on your device to let you know that a project is due tomorrow.

[0133] The server generates a reminder the day before and sends it to the device.

[0134] The device will pop up a reminder on your desktop so you can take action.

[0135] Example of input prompt for generative AI model

[0136] "A user logs into the Desktop Magician system. I want the documents related to the 10 AM meeting to be displayed on the desktop. I also want the system to monitor the user's work activity and warn them if they are working in a different way."

[0137] As described above, the present invention effectively manages a user's schedule through cooperation between a server and a terminal, and provides necessary materials at the appropriate time, thereby improving the user's work efficiency.

[0138] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0139] Step 1: User Login

[0140] Input: User ID and password

[0141] Process: The user starts up the PC and logs in to the Desktop Magician system. The entered user ID and password are sent to the server for authentication.

[0142] Data processing: The server authenticates the user ID and password and generates a success or failure result.

[0143] Output: If authentication is successful, obtain the user's basic information and authorization information and send it to the terminal. If authentication is unsuccessful, display an error message.

[0144] Specific operation: The user enters the user ID and password on the login screen and clicks the login button. The server performs authentication, and if successful, the main screen is displayed.

[0145] Step 2: Retrieving and saving schedule data

[0146] Input: User login information

[0147] Processing: Based on the user's login information, the server uses an API to retrieve schedule data from an external calendar service (e.g., Google Calendar, Outlook Calendar).

[0148] Data processing: The acquired schedule data includes the appointment date, time, title, and details, and is saved in a database on the server.

[0149] Output: Schedule data stored in the database

[0150] What happens: The server sends an API request using the user's calendar account information to retrieve and store schedule data.

[0151] Step 3: Synchronize schedule data

[0152] Input: API request to external calendar service

[0153] Processing: The server periodically sends an API request to the external calendar service to retrieve the latest schedule data.

[0154] Data processing: Compare new or changed schedule data with the database and update it to the latest version.

[0155] Output: Updated database schedule data

[0156] Specific operation: The server makes an API request every hour or at a set interval and updates the database with the retrieved data.

[0157] Step 4: Send schedule data

[0158] Input: Schedule data from the database

[0159] Processing: The terminal periodically sends a request to the server and receives the latest schedule data.

[0160] Data processing: Save the received schedule data to local storage.

[0161] Output: Schedule data in local storage

[0162] Specific operation: The device requests schedule data from the server every 30 minutes and saves the received data.

[0163] Step 5: Automatically display materials

[0164] Input: Local storage schedule data and current time

[0165] Processing: Based on the schedule data, the terminal identifies materials relevant to the current time and displays them on the desktop.

[0166] Data processing: Analyze schedule data and obtain paths to identified materials.

[0167] Output: Documents displayed on the desktop

[0168] Specific operation: The device checks the current time, and if there is a meeting at 10:00, the meeting materials will be displayed on the desktop at 9:45.

[0169] Step 6: Activity monitoring and alerts

[0170] Input: User activity data (active window title bar)

[0171] Action: The device monitors the user's activity and displays a warning pop-up if the user is performing an activity that differs from the scheduled data.

[0172] Data processing: Analyze activity data and compare it with planned work.

[0173] Output: Warning popup

[0174] Specific behavior: If the user is browsing the web in a way that is not part of their schedule, the message "You are currently working on something different from your schedule. Would you like to return to your schedule?" will be displayed.

[0175] Step 7: Reminders

[0176] Input: Schedule data from the database

[0177] Processing: The server analyzes the schedule data and generates reminders when important events or deadlines approach.

[0178] Data processing: Identify important events and deadlines and generate reminder messages.

[0179] Output: The generated reminder is sent to the device and displayed on the desktop.

[0180] What happens: The server generates a reminder for the previous day, and the device displays the reminder on the desktop.

[0181] Example of input prompt for generative AI model

[0182] "A user logs into the Desktop Magician system. I want the documents related to the 10 AM meeting to be displayed on the desktop. I also want the system to monitor the user's work activity and warn them if they are working in a different way."

[0183] (Application example 1)

[0184] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0185] Conventional factory robots did not properly manage work schedules or monitor work status in real time, making it difficult to achieve efficient production activities. Also, when workers performed tasks that deviated from the schedule, appropriate measures could not be taken. Furthermore, the lack of reminders for important tasks and deadlines meant that efficient work operations could not be achieved.

[0186] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0187] In this invention, the server includes a means for acquiring a user's schedule data, a means for identifying necessary documents based on the acquired schedule data, and a means for automatically displaying the identified documents on the desktop of the user's terminal. This enables effective schedule management and real-time monitoring of work status, which was difficult with conventional factory robots. It also enables warnings when work is being performed that deviates from the schedule and reminders for important work and deadlines, thereby achieving efficient work management.

[0188] "User schedule data" refers to information relating to the date, time, schedule contents, etc. that the user has planned in advance.

[0189] "Means of acquisition" refers to application programs or APIs for acquiring user schedule data from external calendar services, etc.

[0190] "Required materials" refers to documents and digital files that are required to perform a specific task based on the schedule data.

[0191] "Means for identification" refers to algorithms or software that analyze acquired schedule data and identify necessary materials based on that data.

[0192] "Means for automatic display" refers to a program or tool for automatically displaying the identified material on the user's terminal.

[0193] A "factory robot" refers to an automated mechanical device designed to perform production tasks in a factory.

[0194] "Current activity" refers to a specific task or activity that a user or factory robot is currently performing.

[0195] "Monitoring means" refers to programs and sensors that observe and record the work activities of users or factory robots in real time.

[0196] The "means for displaying a warning" refers to a program or device for displaying a warning message when a user or a factory robot is performing an activity that differs from the schedule.

[0197] A "reminder" is a message that notifies a user when an important appointment or deadline is approaching, based on the user's schedule data.

[0198] "Means for generating" refers to programs or software for generating reminders based on schedule data.

[0199] "Means for monitoring progress" refers to programs and sensors that monitor the progress of tasks being performed by factory robots in real time.

[0200] "Efficient work instructions" refers to specific instructions and guidance provided to factory robots to help them work more efficiently.

[0201] This invention is a system for improving the work efficiency of factory robots. This system acquires user schedule data, prepares necessary materials and tools based on that data, and provides them to the factory robots at the appropriate time, thereby improving work efficiency.

[0202] System configuration

[0203] The basic system configuration is as follows:

[0204] 1. Server:

[0205] Obtain user schedule data from an external service (e.g., a calendar service).

[0206] The acquired schedule data is managed and saved, and the necessary data is sent to the user's device or factory robot.

[0207] Analyze schedule data and generate reminders.

[0208] 2. Terminal:

[0209] Schedule data provided by the server is received and stored in local storage.

[0210] Display required materials based on the current time.

[0211] 3. Factory robots:

[0212] Automatically prepare materials and tools.

[0213] Monitor work activity and display alerts when schedules deviate.

[0214] Receive reminders and provide work instructions.

[0215] Monitor progress and provide efficient work instructions.

[0216] Overview of program processing

[0217] 1. Acquiring and saving schedule data

[0218] Server: When a user logs in, the server uses an API to retrieve the user's schedule data and saves it in a database. This data includes the date, time, and content of the schedule.

[0219] 2. Schedule data synchronization

[0220] Server: Updates the schedule data at regular intervals to keep it up to date.

[0221] Terminal: Receives schedule data updates from the server at regular intervals and saves them in local storage.

[0222] 3. Automatic display of materials

[0223] Terminal: Based on the current time, schedule data and related materials are identified and displayed on the desktop. For example, if a welding job is scheduled for 10:00 AM, materials related to that job will be displayed on the desktop 15 minutes beforehand.

[0224] 4. Activity Monitoring and Alerts

[0225] Factory robots: Monitor activity and display a warning pop-up after a certain time if the robot is performing work that is different from the schedule, prompting the worker to reconfirm the schedule.

[0226] 5. Reminders

[0227] Server: Analyzes schedule data and generates reminders when important events or deadlines are approaching. Reminders are generated at a set time (e.g., one day before).

[0228] Factory robots: receive reminder notifications and provide work instructions.

[0229] Specific examples

[0230] Consider a scenario in which a specific factory robot is scheduled to perform welding work starting at 10:00 AM. When a user logs into the system in the morning, the server retrieves the user's schedule data from an external calendar service and sends it to the terminal and factory robot. The terminal displays materials related to the 10:00 AM welding work on its desktop, and the factory robot prepares the tools necessary for welding. If the robot is performing a different activity during the work, a warning is displayed and the user is asked to confirm. Furthermore, the server notifies the user that the work deadline is one day away, generates a reminder, and provides instructions to the factory robot.

[0231] Prompt Sentence Examples

[0232] To check the user's morning schedule, retrieve the factory robot's schedule data and display the materials related to the 10:00 a.m. work. If there is any work outside the schedule, display a warning and generate a reminder within 24 hours.

[0233] In this way, the present invention improves the work efficiency of factory robots through specific programs and means, and realizes effective schedule management and real-time work monitoring.

[0234] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0235] Step 1:

[0236] When a user logs in, the server accesses the API of the external calendar service to obtain the user's schedule data. Specifically, it uses the user ID as an API parameter and sends a request. Based on this input, the schedule data (schedule date and time, title, and details) is obtained in JSON format. The obtained data is saved in the server's database.

[0237] Step 2:

[0238] The server updates the schedule data at regular intervals. To update, it accesses the API of the external calendar service again to obtain the latest schedule data. If new data exists, it updates the database contents. This input data keeps the server database up to date with the latest schedule data.

[0239] Step 3:

[0240] The terminal periodically obtains schedule data updates from the server. The terminal receives the latest schedule data from the server and stores the data in the terminal's local storage. This input data allows the terminal to access the latest schedule data.

[0241] Step 4:

[0242] The terminal references the schedule data based on the current time and identifies the relevant materials 15 minutes before the next scheduled appointment. For example, if welding work is scheduled for 10:00 a.m., the materials needed for that work will be identified and displayed on the desktop. Based on this input, the file path of the identified materials is output, and the materials are displayed on the desktop.

[0243] Step 5:

[0244] The factory robot automatically prepares the identified materials and tools. Based on the schedule data, it generates a list of tools required for the day's work and prepares them. For example, in the case of welding work, welding machines and protective equipment are prepared. This input data allows for efficient work preparation.

[0245] Step 6:

[0246] Factory robots use sensors and programs to monitor their work activities. They observe the current work status in real time and compare it with the schedule data. If the work differs from the schedule, a warning pop-up is displayed. This input data detects discrepancies and outputs a warning to the user or robot.

[0247] Step 7:

[0248] The server analyzes the schedule data and generates reminders when important appointments or deadlines are approaching. These reminders are generated at a set time (e.g., one day before) based on the schedule date and time. The generated reminders are sent to the user's device and factory robots. This input data allows the reminders to be output at the appropriate time.

[0249] Step 8:

[0250] The factory robot receives the reminder notification and provides the user with work instructions. For example, a notification is received the day before welding work and an instruction is issued to start preparations the next day. Based on this input data, the reminder content is output and the factory robot is notified reliably.

[0251] This enables effective schedule management, real-time monitoring of work status, and efficient provision of work instructions, which were difficult with conventional factory robots.

[0252] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0253] This invention is a desktop magician system for improving the user's work efficiency and mental well-being. It acquires the user's schedule data and automatically displays necessary documents on the desktop based on that data. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized support.

[0254] System configuration

[0255] The basic system configuration is as follows:

[0256] 1. Server:

[0257] Obtaining user schedule data from an external service (e.g., a calendar API).

[0258] The acquired schedule data is managed and saved, and the necessary data is sent to the user terminal.

[0259] Analyze schedule data and generate reminders.

[0260] 2. Terminal:

[0261] Schedule data provided by the server is received and stored in local storage.

[0262] Display the necessary materials on your desktop based on the current time.

[0263] Monitor users' work activity and display alerts when work is outside of schedule.

[0264] Notify users of reminders.

[0265] It has an emotion engine that recognizes the user's emotions and adjusts desktop displays, alerts, and reminder notifications based on the user's emotional state.

[0266] Overview of program processing

[0267] 1. Acquiring and saving schedule data

[0268] Server: When a user logs in, the server uses an API to retrieve the user's schedule data and saves it in a database. This data includes the date, time, title, and details of the event.

[0269] 2. Schedule data synchronization

[0270] Server: Updates the schedule data at regular intervals to keep it up to date.

[0271] Terminal: Receives schedule data updates from the server at regular intervals and saves them in local storage.

[0272] 3. Automatic display of materials

[0273] Terminal: Based on the current time, schedule data and related materials are identified and displayed on the desktop. For example, if a meeting is scheduled for 10:00 AM, materials related to that meeting will be displayed on the desktop 15 minutes before the meeting.

[0274] 4. Emotion Recognition by Emotion Engine

[0275] Device: While the user is using the PC, the emotion engine processes input data from the built-in camera and microphone and analyzes the user's emotional state in real time. The emotional state is expressed as parameters such as stress level, excitement level, and concentration.

[0276] 5. Activity Monitoring and Alerts

[0277] Terminal: Monitors user activity and displays a warning pop-up after a specified time (e.g., 5 minutes) if the user is performing tasks that differ from the scheduled data. If the user is feeling overly stressed, the emotion engine can soften or postpone the warning.

[0278] 6. Reminders

[0279] Server: Analyzes schedule data and generates reminders when important events or deadlines are approaching. The generated reminders are sent to the device.

[0280] On-device: The emotional engine takes into account the user's emotional state and adjusts the content and timing of reminders. For example, if the user is feeling stressed, the reminder will be more toned down.

[0281] Specific examples

[0282] User's morning activities:

[0283] 1. A user starts up their PC in the morning and logs into Desktop Magician.

[0284] 2. Server: Obtains the user's latest schedule data from an external calendar service and sends it to the device.

[0285] 3. Terminal: Based on the received data, materials related to the meeting at 10:00 AM today are displayed on the desktop.

[0286] Monitoring during work:

[0287] 1. In the morning, the user is browsing the web for an activity other than the scheduled "report writing."

[0288] 2. Terminal: Monitor the title bar of the active window and display a warning popup if it is different from the schedule.

[0289] 3. The popup will say "You are currently working on something different from your schedule. Would you like to return to your schedule?" If the user is feeling annoyed, the warning will be more subtle.

[0290] Emotion engine response:

[0291] 1. When a user continues working for a long time, input from the camera and microphone indicates that stress is high.

[0292] 2. Device: Based on the emotion engine, notify "Take a 5-minute break to relax."

[0293] Reminder:

[0294] 1. In the evening, a reminder appears on your device to let you know that a project is due tomorrow.

[0295] 2. Server: Generates reminders the day before and sends them to the device.

[0296] 3. Device: If the user's stress level is low, the emotion engine will notify them normally, but if it is high, it will notify them in a softer way.

[0297] As described above, the present invention enables users to perform their work efficiently and in a mentally healthy environment through a specific program and its processing that takes into account the user's emotional state.

[0298] The processing flow will be explained below.

[0299] Step 1:

[0300] A user logs in to Desktop Magician. Upon login, the system sends the user's authentication information to the server.

[0301] Step 2:

[0302] The server receives the user's authentication information and verifies that the user is a valid user. If authentication is successful, the server retrieves the user's schedule data from an external API (e.g., a calendar API) and saves it in a database.

[0303] Step 3:

[0304] The server analyzes the schedule data and extracts the user's schedule items (such as the schedule date and time, title, details, and attachment links).

[0305] Step 4:

[0306] The device requests synchronization of schedule data from the server at regular intervals (for example, every morning), and the server sends the latest schedule data to the device.

[0307] Step 5:

[0308] The device saves the received schedule data in local storage. After saving, it compares the current time with the schedule data to identify upcoming events.

[0309] Step 6:

[0310] The device will search for documents related to the specified appointment and automatically display them on the desktop. For example, if a meeting is scheduled for 10:00 a.m., the relevant documents will be displayed on the desktop 15 minutes before the meeting.

[0311] Step 7:

[0312] The device collects data from the camera and microphone built into the user's PC and analyzes it with an emotion engine, which evaluates the user's emotional state (stress, excitement, concentration, etc.) in real time.

[0313] Step 8:

[0314] The device monitors the user's work activity and if the work being performed differs from the scheduled data, a warning pop-up will be displayed after a specified time (e.g., 5 minutes). The content of the warning is adjusted by the emotion engine.

[0315] Step 9:

[0316] The device displays a pop-up warning: "You are currently working on something other than your scheduled schedule. Would you like to return to your scheduled schedule?" If the user's emotional state is stressed, the message will be displayed in a more gentle manner.

[0317] Step 10:

[0318] The server periodically analyzes schedule data and generates reminders when important events or deadlines are approaching, and the generated reminders are sent to the device.

[0319] Step 11:

[0320] Reminders received by the device are displayed as a pop-up on the desktop. The emotion engine adjusts the content and timing of the reminder according to the user's emotional state. For example, if the user is in a high-stress state, the reminder will be displayed at a more discreet time.

[0321] This series of processes enables users to carry out their work efficiently and also to receive support with optimal timing and expression based on their emotional state.

[0322] Example 2

[0323] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0324] Conventional schedule management systems often provide a function to retrieve schedule data and display necessary documents on the user's desktop to improve work efficiency. However, because they do not take the user's emotional state into consideration, excessive work can cause mental strain and reduce work efficiency. In addition, warnings and reminder notifications when the user is working outside of the schedule are displayed in a uniform manner, which makes it difficult to respond flexibly to individual situations.

[0325] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0326] In this invention, the server includes means for acquiring user schedule data, means for identifying necessary materials based on the acquired schedule data, means for automatically displaying the identified materials on the desktop of the user's terminal, means for recognizing the user's emotional state, and means for adjusting the display content of the desktop based on the recognized emotional state. This makes it possible to improve work efficiency and reduce mental burden by taking the user's emotional state into consideration.

[0327] "User schedule data" refers to information including the date, time, title, details, etc. of schedules that a user manages through an external calendar service or manual input.

[0328] The "means of acquisition" refers to a function by which the server collects user schedule data using a means such as an external calendar API.

[0329] The "means for identifying necessary materials" refers to an algorithm or program for identifying related materials or information based on the acquired schedule data.

[0330] The "means for automatic display" is a function for automatically displaying the identified material on the desktop of the user's terminal at the required timing.

[0331] The "means for recognizing the user's emotional state" is a system that uses an emotion analysis algorithm to evaluate the user's emotional state based on data acquired from the built-in camera and microphone.

[0332] The "means for adjusting desktop display content" is a function that dynamically changes the content and timing of information and notifications displayed on the desktop based on the recognized emotional state of the user.

[0333] The "means for monitoring activity" is a system that monitors active windows running on a terminal and user operations, and determines whether or not they are in accordance with a schedule.

[0334] The "means for displaying a warning" is a function that displays a notification or a pop-up to call the user's attention when the user is performing work outside of the schedule.

[0335] The "means for generating reminders" is a function for creating reminders to notify the user of important appointments and deadlines based on schedule data.

[0336] The "means for adjusting the content and timing of reminders" is a system that changes the content of reminder messages and the timing of notifications based on the user's emotional state.

[0337] This invention is a desktop magician system that improves users' work efficiency and mental well-being. It is mainly composed of a server and a user terminal, each of which performs specific functions to achieve performance optimization through schedule management and emotion recognition. Specific implementation methods are described below.

[0338] Basic system configuration

[0339] 1. Server Functions

[0340] Retrieving schedule data: The server retrieves the user's schedule data using the external calendar API. The retrieved data includes the date, time, title, and details of the event.

[0341] Data storage and management: The acquired schedule data is stored in a database on the server and kept accessible upon user request.

[0342] Reminder generation: The server analyzes the schedule data and generates reminders when important appointments or deadlines are approaching, and sends them to the user's terminal.

[0343] 2. Device Features

[0344] Data synchronization and storage: The terminal periodically receives schedule data sent from the server and stores it in local storage.

[0345] Automatic display of necessary materials: Based on the current time, schedule data and related materials are identified and displayed on the desktop. For example, if a meeting is scheduled, materials related to that meeting are automatically displayed.

[0346] Emotion Recognition: The emotion engine analyzes the user's emotional state in real time through the device's built-in camera and microphone. The emotion engine evaluates stress levels, concentration, and other factors and adjusts the content displayed on the desktop accordingly.

[0347] Activity monitoring and warnings: The device monitors the user's work activity and displays a warning pop-up if it detects any deviation from the scheduled activity. It also has the ability to adjust the warning content according to the user's emotional state.

[0348] Reminder notification: Notifies the user of reminders and adjusts the content and timing of notifications using an emotion engine.

[0349] Hardware and software used

[0350] Calendar API: Use an external calendar service (e.g., Google Calendar API) to retrieve schedule data.

[0351] Database: A database system (e.g., MySQL) for storing and managing schedule data on the server.

[0352] Emotion engine: Software that runs deep learning models to analyze a user's facial expressions and voice.

[0353] Built-in camera and microphone: Installed on the user device to collect data for emotion recognition.

[0354] Specific examples

[0355] 1. User's morning activities

[0356] The user starts up the PC and logs into Desktop Magician.

[0357] Server: Obtains the latest schedule data from an external calendar service and sends it to the device.

[0358] Device: Display materials related to the 10:00 AM meeting on your desktop.

[0359] 2. Monitoring during work

[0360] The user is browsing the web for a purpose other than "writing reports."

[0361] On your device: Display a warning popup saying "You are currently working on something other than your scheduled schedule. Would you like to return to your scheduled schedule?"

[0362] 3. Emotional engine response

[0363] When working for long periods of time, input from the camera and microphone is used to determine high stress levels.

[0364] Device: Display a notification that says "Take a 5-minute break to relax."

[0365] 4. Reminders

[0366] In the evening, a reminder appears on your device to let you know that a project is due tomorrow.

[0367] Server: Generates reminders and sends them to devices.

[0368] Device: If stress is low, notifications will be sent as normal, but if stress is high, notifications will be sent in a softer tone.

[0369] Prompt Sentence Examples

[0370] Prompt Sentence Examples

[0371] "Please explain in natural language the specific processing steps of the Desktop Magician system. Include the following items: obtaining user schedule data, automatically displaying materials, emotion recognition using an emotion engine, activity monitoring and alerts, and reminder notifications. Please clearly indicate the names and functions of the specific hardware and software used in all processing steps."

[0372] Specific questions

[0373] "After a user logs in, how does the server get the user's schedule data? And how is that data processed?"

[0374] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0375] Program processing flow and specific explanation

[0376] Step 1: User Login

[0377] User: Start the PC, click on the Desktop Magician app to go to the login screen, enter your username and password, and click the "Login" button.

[0378] Terminal: Sends the entered authentication information to the server.

[0379] Server: Compares the user information stored in the database, and if authentication is successful, generates an authentication token and sends it to the terminal.

[0380] Input: Username, Password

[0381] Output: Authentication token

[0382] Step 2: Retrieving and saving schedule data

[0383] Server: If authentication is successful, the external calendar API is used to retrieve the user's schedule data, including the event date, time, title, and details.

[0384] Server: Analyzes the acquired schedule data and stores it in a database.

[0385] Input: Schedule data from an external calendar API

[0386] Output: Schedule data stored in the server database

[0387] Step 3: Synchronize schedule data

[0388] Terminal: The Desktop Magician application running on the user's PC sends a schedule data update request to the server at regular intervals (e.g., every 5 minutes).

[0389] Server: Sends the latest schedule data to the terminal.

[0390] Terminal: Save the received schedule data in local storage.

[0391] Input: Schedule data update request from the server

[0392] Output: The latest schedule data saved in local storage

[0393] Step 4: Automatically display required materials

[0394] Device: Check your next appointment based on the current time. For example, if you have a meeting scheduled for 10:00 AM, identify the materials related to that meeting at 9:45 AM.

[0395] Device: The identified document will be displayed as a pop-up in a prominent location on your desktop.

[0396] Input: Schedule data in local storage

[0397] Output: Related documents displayed on your desktop

[0398] Step 5: Emotion Recognition

[0399] Device: Captures the user's facial expressions and voice in real time through the built-in camera and microphone.

[0400] Emotion engine: Analyzes acquired data using a deep learning model to evaluate the user's emotional state.

[0401] Device: Dynamically adjusts desktop display content and notification timing based on evaluation results.

[0402] Input: Data from the built-in camera and microphone

[0403] Output: Evaluated user emotional state and adjusted desktop display content

[0404] Step 6: Activity monitoring and alerts

[0405] Terminal: Monitors the title bar of the active window to see if the user is working according to a schedule.

[0406] Terminal: If the user is working outside of their schedule, a warning popup will appear after a specified time (e.g. 5 minutes).

[0407] Emotion engine: If the user's emotional state is high stress, the warning content will be softened.

[0408] Input: Active window title bar information

[0409] Output: Warning popup content

[0410] Step 7: Reminders

[0411] Server: Analyzes schedule data and generates reminders when important appointments or deadlines are approaching.

[0412] Server: Sends the generated reminders to the device.

[0413] Device: Notifies the user of received reminders and adjusts the content and timing using the emotion engine. For example, it sends a subtle notification such as "Remember the project submission deadline tomorrow."

[0414] Input: Schedule data

[0415] Output: Reminders notified to the user

[0416] (Application example 2)

[0417] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0418] In logistics center operations, managing schedules and related documents is essential for employees to work efficiently. However, doing this manually takes time and is prone to errors, which can lead to reduced work efficiency. Furthermore, long working hours and a stressful work environment can also put employees at risk of suffering from mental health issues, so a solution to these issues is needed.

[0419] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring user schedule data, means for identifying necessary materials based on the acquired schedule data, means for automatically displaying the identified materials on a display device on the user's terminal, means for recognizing the user's emotion, and means for adjusting the display content based on the recognized emotion. This allows employees to proceed with their work based on the automatically displayed schedule and related materials, and the use of emotion recognition technology enables them to work while maintaining their mental health.

[0420] "User schedule data" is information about the user's work schedule and task times.

[0421] "Means of acquisition" refers to the methods and systems used to collect schedule data from external databases and APIs.

[0422] "Necessary materials" refers to documents and information related to schedule data, such as work instructions and picking lists, which are necessary for carrying out work.

[0423] "Means for identifying" refers to a method or system that analyzes schedule data to determine what materials are needed at that time.

[0424] "User's Device" refers to a computer, smartphone, smart glasses, or other electronic device used by a User to conduct business.

[0425] A "display device" is a screen or display that allows a user to visually view information.

[0426] "Means for automatic display" is a mechanism for automatically displaying necessary materials on the display device of a terminal under certain conditions.

[0427] "Means for recognizing emotions" refers to technologies or systems that analyze the user's facial expressions and vocal characteristics to infer their emotional state at that time.

[0428] The "means for adjusting the displayed content" refers to a method or system for changing the type of information displayed or the way it is expressed based on the recognized emotion.

[0429] The "activity monitoring means" is a system for observing and recording the user's work status in real time.

[0430] The "means for displaying warnings" is a mechanism for displaying warning messages on the terminal display when the user's current activity deviates from the schedule or when a problem occurs.

[0431] The "means for generating reminders" is a notification function that notifies the user of important appointments and task deadlines based on schedule data.

[0432] The "means for displaying a reminder" is a system for displaying the generated reminder on the display device of the user's terminal.

[0433] "Means for adjusting reminder content and timing" refers to a system or method for changing the content of reminder messages and / or notification timing in consideration of a perceived emotional state.

[0434] This invention provides a "Logistics Magician" system for improving the work efficiency and mental health of employees at logistics centers. This system acquires the user's schedule data and automatically presents relevant materials based on that data, thereby improving work efficiency. It also uses emotion recognition technology to support the mental health of employees.

[0435] Basic system configuration

[0436] The basic configuration of the system is as follows:

[0437] 1. Server:

[0438] Obtaining schedule data: The server uses the API to obtain the user's schedule data from an external database.

[0439] Data management and storage: The acquired schedule data is stored in a database and the necessary data is sent to the terminal.

[0440] Generate reminders: Analyze schedule data and generate reminders.

[0441] 2. Terminal:

[0442] Schedule data synchronization: The terminal receives schedule data provided by the server and stores it in local storage.

[0443] Automatic display of materials: Based on the current time, schedule data and related materials are identified and displayed on the display device.

[0444] Activity monitoring and alerts: Monitors users' work activity and displays alerts if it deviates from the schedule.

[0445] Emotion engine: Using emotion recognition technology, it analyzes the user's emotional state and adjusts the display content and reminder content.

[0446] Specific processing explanation

[0447] 1. Data acquisition and display:

[0448] Hardware: Smart glasses, smartphones

[0449] The server retrieves the user's schedule data from an external database or API and stores it in the database. The device receives the latest schedule data from the server at regular intervals and stores it in local storage. When executed, the schedule data and related materials are displayed on the display device based on the current time.

[0450] Example: If a user has a meeting scheduled at 10:00 AM, materials related to that meeting will automatically appear on the display at 9:45 AM.

[0451] 2. Emotion recognition:

[0452] Software: OpenCV

[0453] The system analyzes the device's camera footage in real time and estimates the user's emotional state from their facial expressions, thereby assessing their stress level and fatigue.

[0454] Example: If the user feels fatigued, the device will notify them to "take a 5-minute break to relax."

[0455] 3. Notifications and Reminders:

[0456] It notifies users of important tasks and break times based on schedule data, and adjusts the content and timing of reminder messages based on emotional state.

[0457] Example: When a user is in a high-stress state, a reminder is displayed discreetly: "It's time for your next task. Please be mindful."

[0458] Prompt Sentence Examples

[0459] "If the user is not performing the current task accurately, display 'Are you continuing to work as planned?' and generate a prompt sentence recommending a break based on the emotion recognition results."

[0460] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0461] Step 1:

[0462] Acquiring and managing schedule data

[0463] Server: When a user logs in, the server uses an API to retrieve the user's schedule data from an external database, including the date, time, title, and details of the event.

[0464] Input: User ID, API authentication information.

[0465] Data processing: Call the API and receive schedule data in JSON format as a response.

[0466] Output: Schedule data in JSON format.

[0467] What happens: The server runs this process periodically and saves the schedule data to a database to keep it up to date.

[0468] Step 2:

[0469] Schedule data synchronization

[0470] Terminal: The terminal receives schedule data provided by the server at regular intervals and stores it in local storage.

[0471] Input: The request to the server.

[0472] Data calculation: Analyzes the JSON data sent from the server and saves it in local storage.

[0473] Output: The latest schedule data is saved locally.

[0474] Specific operation: The device sends a request to the server every five minutes to obtain the latest schedule data and update it to local storage.

[0475] Step 3:

[0476] Automatic display of materials

[0477] Terminal: Based on the current time, schedule data and related materials are identified and displayed on the display device.

[0478] Input: Local storage schedule data, current time.

[0479] Data calculation: Compare the current time with schedule data to identify the corresponding materials.

[0480] Output: Relevant material displayed on a display device.

[0481] What it does: If you have a meeting scheduled for 10:00 AM, presentations and documents related to that meeting will automatically appear at 9:45 AM.

[0482] Step 4:

[0483] emotion recognition

[0484] Device: Analyzes camera images from smart glasses or smartphones in real time and estimates the user's emotional state from their facial expressions.

[0485] Input: Camera footage.

[0486] Data calculation: Using OpenCV, emotions are estimated using a facial expression analysis model.

[0487] Output: Emotional state, such as stress level and fatigue level.

[0488] What it does: The device's camera captures the user's face and analyzes the image in real time to determine whether the user is tired.

[0489] Step 5:

[0490] Generate and coordinate notifications and reminders

[0491] Server (generation) and device (adjustment): The server generates reminders based on schedule data and sends them to the device. The device adjusts the content and timing of the reminders based on the recognized emotions.

[0492] Input: Schedule data, emotional state.

[0493] Data calculation: Analyze schedule data and adjust reminders based on emotional state.

[0494] Output: The adjusted reminder is displayed on the display device.

[0495] What it does: If the user is in a high-stress state, the reminder is subtly displayed as "It's time for your next task. Please be mindful."

[0496] Step 6:

[0497] Activity monitoring and alerts

[0498] Terminal: Monitors the user's work activity and displays a warning if it differs from the scheduled data.

[0499] Input: Current activity, schedule data.

[0500] Data calculation: Compares activity content with schedule data and generates a warning if there is a discrepancy.

[0501] Output: A warning message is displayed.

[0502] What it does: If a user is browsing a website other than the scheduled task, a message will pop up asking, "Are you still working as planned?"

[0503] Prompt Sentence Examples

[0504] "If the user is not performing the current task accurately, display 'Are you continuing to work as planned?' and generate a prompt sentence recommending a break based on the emotion recognition results."

[0505] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0506] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0507] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0508] [Second embodiment]

[0509] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0510] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0511] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0512] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0513] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0514] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0515] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0516] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0517] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0518] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0519] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0520] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0521] This invention is a desktop magician system for improving the user's work efficiency. It aims to acquire the user's schedule data and automatically display the necessary documents on the desktop based on that data. The system is realized by linking a server and a terminal.

[0522] System configuration

[0523] The basic system configuration is as follows:

[0524] 1. Server:

[0525] Retrieve user schedule data from external services (e.g., Google Calendar or Outlook Calendar).

[0526] The acquired schedule data is managed and saved, and the necessary data is sent to the user terminal.

[0527] Analyze schedule data and generate reminders.

[0528] 2. Terminal:

[0529] Schedule data provided by the server is received and stored in local storage.

[0530] Display the necessary materials on your desktop based on the current time.

[0531] Monitor users' work activity and display alerts when work is outside of schedule.

[0532] Notify users of reminders.

[0533] Overview of program processing

[0534] 1. Acquiring and saving schedule data

[0535] Server: When a user logs in, the server uses an API to retrieve the user's schedule data and saves it in a database. This data includes the date, time, title, and details of the event.

[0536] 2. Schedule data synchronization

[0537] Server: Updates the schedule data at regular intervals to keep it up to date.

[0538] Terminal: Receives schedule data updates from the server at regular intervals and saves them in local storage.

[0539] 3. Automatic display of materials

[0540] Terminal: Based on the current time, schedule data and related materials are identified and displayed on the desktop. For example, if a meeting is scheduled for 10:00 AM, materials related to that meeting will be displayed on the desktop 15 minutes before the meeting.

[0541] 4. Activity Monitoring and Alerts

[0542] Terminal: Monitors user activity and displays a warning pop-up after a certain period of time if the user is performing tasks that differ from the scheduled data. This pop-up prompts the user to recheck their schedule.

[0543] 5. Reminders

[0544] Server: Analyzes schedule data and generates reminders when important events or deadlines are approaching. Reminders are generated at set times (e.g., 3 days before, 1 day before).

[0545] Device: Reminder notifications are received and displayed on the desktop. After the user confirms the reminder, they can also use the snooze function to be notified again.

[0546] Specific examples

[0547] User's morning activities:

[0548] 1. A user starts up their PC in the morning and logs into Desktop Magician.

[0549] 2. Server: Obtains the user's latest schedule data from an external calendar service and sends it to the device.

[0550] 3. Terminal: Based on the received data, materials related to the meeting at 10:00 AM today are displayed on the desktop.

[0551] Monitoring during work:

[0552] 1. In the morning, the user is browsing the web for an activity other than the scheduled "report writing."

[0553] 2. Terminal: Monitor the title bar of the active window and display a warning popup if it is different from the schedule.

[0554] 3. A pop-up will appear saying, "You are currently working on something different from your schedule. Would you like to return to your schedule?" and the user will select "Yes."

[0555] Reminder:

[0556] 1. In the evening, a reminder appears on your device to let you know that a project is due tomorrow.

[0557] 2. Server: Generates reminders the day before and sends them to the device.

[0558] 3. Device: Reminders pop up on the desktop so users can take action.

[0559] As described above, the present invention provides an environment in which a user can work efficiently through a specific program and its processing.

[0560] The processing flow will be explained below.

[0561] Step 1:

[0562] The user logs in to Desktop Magician. Once logged in, the system sends the user authentication information to the server.

[0563] Step 2:

[0564] The server receives the user's authentication information and verifies that the user is a valid user. If authentication is successful, the server retrieves the user's schedule data (e.g., Google Calendar or Outlook Calendar) from an external API.

[0565] Step 3:

[0566] The server analyzes the acquired schedule data, extracts the user's schedule items, and stores them in a database, including the schedule date and time, title, details, and links to attachments.

[0567] Step 4:

[0568] The device requests synchronization of schedule data from the server at regular intervals (for example, every morning), and the server sends the latest schedule data to the device.

[0569] Step 5:

[0570] The device stores the received schedule data in local storage, then compares the current time with the schedule data to identify events scheduled within the next few hours.

[0571] Step 6:

[0572] The device will search for materials related to the identified event and automatically display them on the desktop. For example, if a meeting is scheduled for 10:00 a.m., materials related to this meeting will be displayed on the desktop 15 minutes before the meeting.

[0573] Step 7:

[0574] The device monitors the user's work activity and displays a warning popup after a specified time (e.g., 5 minutes) if the user is not performing the scheduled task.

[0575] Step 8:

[0576] The device will display a pop-up warning to the user, saying, "You are currently working on something different from your scheduled schedule. Would you like to return to your scheduled schedule?" If the user selects "Yes," they will be provided with a link to easily return to their scheduled materials or applications.

[0577] Step 9:

[0578] The server periodically analyzes schedule data and generates reminders when important events or deadlines are approaching, which are then sent to the device.

[0579] Step 10:

[0580] The device will display the received reminder as a pop-up on the desktop, and after the user confirms the reminder, they can set the snooze function to be notified again later.

[0581] This series of processes allows users to concentrate on their work efficiently and makes schedule management easier.

[0582] Example 1

[0583] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0584] Conventional schedule management systems require users to prepare materials themselves, and do not provide sufficient warnings or reminders when they are working outside of the schedule, which can lead to reduced work efficiency.

[0585] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0586] In this invention, the server includes means for acquiring user schedule data from an external service, means for saving the acquired schedule data in a database, and means for periodically updating the schedule data to keep it up to date. This allows necessary materials to be automatically displayed on the desktop based on the latest schedule data, improving the user's work efficiency.

[0587] The terminal also includes means for identifying materials relevant to the current time based on the schedule data and automatically displaying them on the desktop, means for monitoring the user's activity during work and displaying a warning if the user is performing work that differs from the schedule, means for analyzing the schedule data and generating a reminder when an important appointment or deadline approaches, and means for displaying the generated reminder on the desktop. This enables the user to accurately perform the scheduled work and proceed with work without missing important appointments or deadlines.

[0588] "User schedule data" refers to information about plans and events that a user has registered in an external service (for example, a calendar service), and includes information such as date and time, title, and details.

[0589] "External service" refers to an online platform for managing, storing, and providing users' schedule data, such as a calendar application or a cloud-based schedule management service.

[0590] A "database" is a system that organizes, stores, and manages acquired schedule data.

[0591] "Regularly updated" means that schedule data is reacquired from an external service at regular intervals to keep the contents of the database up to date.

[0592] "Necessary materials" refers to documents and files related to scheduled tasks and events based on the user's schedule data.

[0593] "Automatically displaying on the desktop" means displaying materials on the screen of the user's terminal without the need for manual operation.

[0594] "Terminal" refers to an electronic computer that allows a user to use the Desktop Magician System, and includes desktop PCs, notebook PCs, tablets, etc.

[0595] "Activity" refers to the work or operation that a user performs on a terminal, and indicates the type of work that can be determined, for example, by the title bar of an active window.

[0596] A "warning popup" is a small window that displays a warning or instructions when the user is performing work that is different from the schedule.

[0597] A "reminder" is a message that notifies the user when an important event or deadline is approaching, and is generated at a specified timing.

[0598] "Local storage" refers to a storage device (such as a hard disk or SSD) built into the user's terminal, and is used to store schedule data and related materials received from the server.

[0599] This invention is a desktop magician system for improving the user's work efficiency. The system acquires the user's schedule data and automatically displays the necessary documents on the desktop based on that data. The system is realized by linking a server and a terminal.

[0600] server

[0601] When a user logs in to the Desktop Magician system, the server uses an API to retrieve the user's schedule data from external services (such as Google Calendar or Outlook Calendar). This retrieved data includes the date, time, title, and details of the appointment. The retrieved data is then stored in a database on the server. The server then updates the schedule data at regular intervals to keep it up to date. In addition, the server analyzes the schedule data and generates reminders when important appointments or deadlines are approaching, which are then sent to the device.

[0602] Terminal

[0603] The device periodically receives schedule data provided by the server and saves it in local storage. Based on the current time, it identifies the schedule data and related materials and automatically displays them on the desktop. The device also monitors the user's work activity and displays a warning pop-up after a certain period of time if the user is performing work that differs from the schedule data. This pop-up prompts the user to reconfirm their schedule. It also sends reminder notifications, providing information for the user to take action. Reminder notifications are displayed as a pop-up on the desktop, and after the user confirms the reminder, they can use the snooze function to be notified again.

[0604] Usage example

[0605] User's morning activities:

[0606] A user starts up their PC in the morning and logs into Desktop Magician.

[0607] The server obtains the user's latest schedule data from an external calendar service and sends it to the terminal.

[0608] Based on the data received by the device, materials related to the meeting at 10:00 a.m. today will be displayed on the desktop.

[0609] Monitoring during work:

[0610] In the morning, the user is browsing the web, which is different from the scheduled "report writing" activity.

[0611] The terminal monitors the title bar of the active window and displays a warning popup if it is different from what is expected.

[0612] The popup displays "You are currently working on something different from your schedule. Would you like to return to your schedule?" and the user selects "Yes."

[0613] Reminder:

[0614] In the evening, a reminder appears on your device to let you know that a project is due tomorrow.

[0615] The server generates a reminder the day before and sends it to the device.

[0616] The device will pop up a reminder on your desktop so you can take action.

[0617] Example of input prompt for generative AI model

[0618] "A user logs into the Desktop Magician system. I want the documents related to the 10 AM meeting to be displayed on the desktop. I also want the system to monitor the user's work activity and warn them if they are working in a different way."

[0619] As described above, the present invention effectively manages a user's schedule through cooperation between a server and a terminal, and provides necessary materials at the appropriate time, thereby improving the user's work efficiency.

[0620] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0621] Step 1: User Login

[0622] Input: User ID and password

[0623] Process: The user starts up the PC and logs in to the Desktop Magician system. The entered user ID and password are sent to the server for authentication.

[0624] Data processing: The server authenticates the user ID and password and generates a success or failure result.

[0625] Output: If authentication is successful, obtain the user's basic information and authorization information and send it to the terminal. If authentication is unsuccessful, display an error message.

[0626] Specific operation: The user enters the user ID and password on the login screen and clicks the login button. The server performs authentication, and if successful, the main screen is displayed.

[0627] Step 2: Retrieving and saving schedule data

[0628] Input: User login information

[0629] Processing: Based on the user's login information, the server uses an API to retrieve schedule data from an external calendar service (e.g., Google Calendar, Outlook Calendar).

[0630] Data processing: The acquired schedule data includes the appointment date, time, title, and details, and is saved in a database on the server.

[0631] Output: Schedule data stored in the database

[0632] What happens: The server sends an API request using the user's calendar account information to retrieve and store schedule data.

[0633] Step 3: Synchronize schedule data

[0634] Input: API request to external calendar service

[0635] Processing: The server periodically sends an API request to the external calendar service to retrieve the latest schedule data.

[0636] Data processing: Compare new or changed schedule data with the database and update it to the latest version.

[0637] Output: Updated database schedule data

[0638] Specific operation: The server makes an API request every hour or at a set interval and updates the database with the retrieved data.

[0639] Step 4: Send schedule data

[0640] Input: Schedule data from the database

[0641] Processing: The terminal periodically sends a request to the server and receives the latest schedule data.

[0642] Data processing: Save the received schedule data to local storage.

[0643] Output: Schedule data in local storage

[0644] Specific operation: The device requests schedule data from the server every 30 minutes and saves the received data.

[0645] Step 5: Automatically display materials

[0646] Input: Local storage schedule data and current time

[0647] Processing: Based on the schedule data, the terminal identifies materials relevant to the current time and displays them on the desktop.

[0648] Data processing: Analyze schedule data and obtain paths to identified materials.

[0649] Output: Documents displayed on the desktop

[0650] Specific operation: The device checks the current time, and if there is a meeting at 10:00, the meeting materials will be displayed on the desktop at 9:45.

[0651] Step 6: Activity monitoring and alerts

[0652] Input: User activity data (active window title bar)

[0653] Action: The device monitors the user's activity and displays a warning pop-up if the user is performing an activity that differs from the scheduled data.

[0654] Data processing: Analyze activity data and compare it with planned work.

[0655] Output: Warning popup

[0656] Specific behavior: If the user is browsing the web in a way that is not part of their schedule, the message "You are currently working on something different from your schedule. Would you like to return to your schedule?" will be displayed.

[0657] Step 7: Reminders

[0658] Input: Schedule data from the database

[0659] Processing: The server analyzes the schedule data and generates reminders when important events or deadlines approach.

[0660] Data processing: Identify important events and deadlines and generate reminder messages.

[0661] Output: The generated reminder is sent to the device and displayed on the desktop.

[0662] What happens: The server generates a reminder for the previous day, and the device displays the reminder on the desktop.

[0663] Example of input prompt for generative AI model

[0664] "A user logs into the Desktop Magician system. I want the documents related to the 10 AM meeting to be displayed on the desktop. I also want the system to monitor the user's work activity and warn them if they are working in a different way."

[0665] (Application example 1)

[0666] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0667] Conventional factory robots did not properly manage work schedules or monitor work status in real time, making it difficult to achieve efficient production activities. Also, when workers performed tasks that deviated from the schedule, appropriate measures could not be taken. Furthermore, the lack of reminders for important tasks and deadlines meant that efficient work operations could not be achieved.

[0668] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0669] In this invention, the server includes a means for acquiring a user's schedule data, a means for identifying necessary documents based on the acquired schedule data, and a means for automatically displaying the identified documents on the desktop of the user's terminal. This enables effective schedule management and real-time monitoring of work status, which was difficult with conventional factory robots. It also enables warnings when work is being performed that deviates from the schedule and reminders for important work and deadlines, thereby achieving efficient work management.

[0670] "User schedule data" refers to information relating to the date, time, schedule contents, etc. that the user has planned in advance.

[0671] "Means of acquisition" refers to application programs or APIs for acquiring user schedule data from external calendar services, etc.

[0672] "Required materials" refers to documents and digital files that are required to perform a specific task based on the schedule data.

[0673] "Means for identification" refers to algorithms or software that analyze acquired schedule data and identify necessary materials based on that data.

[0674] "Means for automatic display" refers to a program or tool for automatically displaying the identified material on the user's terminal.

[0675] A "factory robot" refers to an automated mechanical device designed to perform production tasks in a factory.

[0676] "Current activity" refers to a specific task or activity that a user or factory robot is currently performing.

[0677] "Monitoring means" refers to programs and sensors that observe and record the work activities of users or factory robots in real time.

[0678] The "means for displaying a warning" refers to a program or device for displaying a warning message when a user or a factory robot is performing an activity that differs from the schedule.

[0679] A "reminder" is a message that notifies a user when an important appointment or deadline is approaching, based on the user's schedule data.

[0680] "Means for generating" refers to programs or software for generating reminders based on schedule data.

[0681] "Means for monitoring progress" refers to programs and sensors that monitor the progress of tasks being performed by factory robots in real time.

[0682] "Efficient work instructions" refers to specific instructions and guidance provided to factory robots to help them work more efficiently.

[0683] This invention is a system for improving the work efficiency of factory robots. This system acquires user schedule data, prepares necessary materials and tools based on that data, and provides them to the factory robots at the appropriate time, thereby improving work efficiency.

[0684] System configuration

[0685] The basic system configuration is as follows:

[0686] 1. Server:

[0687] Obtain user schedule data from an external service (e.g., a calendar service).

[0688] The acquired schedule data is managed and saved, and the necessary data is sent to the user's device or factory robot.

[0689] Analyze schedule data and generate reminders.

[0690] 2. Terminal:

[0691] Schedule data provided by the server is received and stored in local storage.

[0692] Display required materials based on the current time.

[0693] 3. Factory robots:

[0694] Automatically prepare materials and tools.

[0695] Monitor work activity and display alerts when schedules deviate.

[0696] Receive reminders and provide work instructions.

[0697] Monitor progress and provide efficient work instructions.

[0698] Overview of program processing

[0699] 1. Acquiring and saving schedule data

[0700] Server: When a user logs in, the server uses an API to retrieve the user's schedule data and saves it in a database. This data includes the date, time, and content of the schedule.

[0701] 2. Schedule data synchronization

[0702] Server: Updates the schedule data at regular intervals to keep it up to date.

[0703] Terminal: Receives schedule data updates from the server at regular intervals and saves them in local storage.

[0704] 3. Automatic display of materials

[0705] Terminal: Based on the current time, schedule data and related materials are identified and displayed on the desktop. For example, if a welding job is scheduled for 10:00 AM, materials related to that job will be displayed on the desktop 15 minutes beforehand.

[0706] 4. Activity Monitoring and Alerts

[0707] Factory robots: Monitor activity and display a warning pop-up after a certain time if the robot is performing work that is different from the schedule, prompting the worker to reconfirm the schedule.

[0708] 5. Reminders

[0709] Server: Analyzes schedule data and generates reminders when important events or deadlines are approaching. Reminders are generated at a set time (e.g., one day before).

[0710] Factory robots: receive reminder notifications and provide work instructions.

[0711] Specific examples

[0712] Consider a scenario in which a specific factory robot is scheduled to perform welding work starting at 10:00 AM. When a user logs into the system in the morning, the server retrieves the user's schedule data from an external calendar service and sends it to the terminal and factory robot. The terminal displays materials related to the 10:00 AM welding work on its desktop, and the factory robot prepares the tools necessary for welding. If the robot is performing a different activity during the work, a warning is displayed and the user is asked to confirm. Furthermore, the server notifies the user that the work deadline is one day away, generates a reminder, and provides instructions to the factory robot.

[0713] Prompt Sentence Examples

[0714] To check the user's morning schedule, retrieve the factory robot's schedule data and display the materials related to the 10:00 a.m. work. If there is any work outside the schedule, display a warning and generate a reminder within 24 hours.

[0715] In this way, the present invention improves the work efficiency of factory robots through specific programs and means, and realizes effective schedule management and real-time work monitoring.

[0716] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0717] Step 1:

[0718] When a user logs in, the server accesses the API of the external calendar service to obtain the user's schedule data. Specifically, it uses the user ID as an API parameter and sends a request. Based on this input, the schedule data (schedule date and time, title, and details) is obtained in JSON format. The obtained data is saved in the server's database.

[0719] Step 2:

[0720] The server updates the schedule data at regular intervals. To update, it accesses the API of the external calendar service again to obtain the latest schedule data. If new data exists, it updates the database contents. This input data keeps the server database up to date with the latest schedule data.

[0721] Step 3:

[0722] The terminal periodically obtains schedule data updates from the server. The terminal receives the latest schedule data from the server and stores the data in the terminal's local storage. This input data allows the terminal to access the latest schedule data.

[0723] Step 4:

[0724] The terminal references the schedule data based on the current time and identifies the relevant materials 15 minutes before the next scheduled appointment. For example, if welding work is scheduled for 10:00 a.m., the materials needed for that work will be identified and displayed on the desktop. Based on this input, the file path of the identified materials is output, and the materials are displayed on the desktop.

[0725] Step 5:

[0726] The factory robot automatically prepares the identified materials and tools. Based on the schedule data, it generates a list of tools required for the day's work and prepares them. For example, in the case of welding work, welding machines and protective equipment are prepared. This input data allows for efficient work preparation.

[0727] Step 6:

[0728] Factory robots use sensors and programs to monitor their work activities. They observe the current work status in real time and compare it with the schedule data. If the work differs from the schedule, a warning pop-up is displayed. This input data detects discrepancies and outputs a warning to the user or robot.

[0729] Step 7:

[0730] The server analyzes the schedule data and generates reminders when important appointments or deadlines are approaching. These reminders are generated at a set time (e.g., one day before) based on the schedule date and time. The generated reminders are sent to the user's device and factory robots. This input data allows the reminders to be output at the appropriate time.

[0731] Step 8:

[0732] The factory robot receives the reminder notification and provides the user with work instructions. For example, a notification is received the day before welding work and an instruction is issued to start preparations the next day. Based on this input data, the reminder content is output and the factory robot is notified reliably.

[0733] This enables effective schedule management, real-time monitoring of work status, and efficient provision of work instructions, which were difficult with conventional factory robots.

[0734] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0735] This invention is a desktop magician system for improving the user's work efficiency and mental well-being. It acquires the user's schedule data and automatically displays necessary documents on the desktop based on that data. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized support.

[0736] System configuration

[0737] The basic system configuration is as follows:

[0738] 1. Server:

[0739] Obtaining user schedule data from an external service (e.g., a calendar API).

[0740] The acquired schedule data is managed and saved, and the necessary data is sent to the user terminal.

[0741] Analyze schedule data and generate reminders.

[0742] 2. Terminal:

[0743] Schedule data provided by the server is received and stored in local storage.

[0744] Display the necessary materials on your desktop based on the current time.

[0745] Monitor users' work activity and display alerts when work is outside of schedule.

[0746] Notify users of reminders.

[0747] It has an emotion engine that recognizes the user's emotions and adjusts desktop displays, alerts, and reminder notifications based on the user's emotional state.

[0748] Overview of program processing

[0749] 1. Acquiring and saving schedule data

[0750] Server: When a user logs in, the server uses an API to retrieve the user's schedule data and saves it in a database. This data includes the date, time, title, and details of the event.

[0751] 2. Schedule data synchronization

[0752] Server: Updates the schedule data at regular intervals to keep it up to date.

[0753] Terminal: Receives schedule data updates from the server at regular intervals and saves them in local storage.

[0754] 3. Automatic display of materials

[0755] Terminal: Based on the current time, schedule data and related materials are identified and displayed on the desktop. For example, if a meeting is scheduled for 10:00 AM, materials related to that meeting will be displayed on the desktop 15 minutes before the meeting.

[0756] 4. Emotion Recognition by Emotion Engine

[0757] Device: While the user is using the PC, the emotion engine processes input data from the built-in camera and microphone and analyzes the user's emotional state in real time. The emotional state is expressed as parameters such as stress level, excitement level, and concentration.

[0758] 5. Activity Monitoring and Alerts

[0759] Terminal: Monitors user activity and displays a warning pop-up after a specified time (e.g., 5 minutes) if the user is performing tasks that differ from the scheduled data. If the user is feeling overly stressed, the emotion engine can soften or postpone the warning.

[0760] 6. Reminders

[0761] Server: Analyzes schedule data and generates reminders when important events or deadlines are approaching. The generated reminders are sent to the device.

[0762] On-device: The emotional engine takes into account the user's emotional state and adjusts the content and timing of reminders. For example, if the user is feeling stressed, the reminder will be more toned down.

[0763] Specific examples

[0764] User's morning activities:

[0765] 1. A user starts up their PC in the morning and logs into Desktop Magician.

[0766] 2. Server: Obtains the user's latest schedule data from an external calendar service and sends it to the device.

[0767] 3. Terminal: Based on the received data, materials related to the meeting at 10:00 AM today are displayed on the desktop.

[0768] Monitoring during work:

[0769] 1. In the morning, the user is browsing the web for an activity other than the scheduled "report writing."

[0770] 2. Terminal: Monitor the title bar of the active window and display a warning popup if it is different from the schedule.

[0771] 3. The popup will say "You are currently working on something different from your schedule. Would you like to return to your schedule?" If the user is feeling annoyed, the warning will be more subtle.

[0772] Emotion engine response:

[0773] 1. When a user continues working for a long time, input from the camera and microphone indicates that stress is high.

[0774] 2. Device: Based on the emotion engine, notify "Take a 5-minute break to relax."

[0775] Reminder:

[0776] 1. In the evening, a reminder appears on your device to let you know that a project is due tomorrow.

[0777] 2. Server: Generates reminders the day before and sends them to the device.

[0778] 3. Device: If the user's stress level is low, the emotion engine will notify them normally, but if it is high, it will notify them in a softer way.

[0779] As described above, the present invention enables users to perform their work efficiently and in a mentally healthy environment through a specific program and its processing that takes into account the user's emotional state.

[0780] The processing flow will be explained below.

[0781] Step 1:

[0782] A user logs in to Desktop Magician. Upon login, the system sends the user's authentication information to the server.

[0783] Step 2:

[0784] The server receives the user's authentication information and verifies that the user is a valid user. If authentication is successful, the server retrieves the user's schedule data from an external API (e.g., a calendar API) and saves it in a database.

[0785] Step 3:

[0786] The server analyzes the schedule data and extracts the user's schedule items (such as the schedule date and time, title, details, and attachment links).

[0787] Step 4:

[0788] The device requests synchronization of schedule data from the server at regular intervals (for example, every morning), and the server sends the latest schedule data to the device.

[0789] Step 5:

[0790] The device saves the received schedule data in local storage. After saving, it compares the current time with the schedule data to identify upcoming events.

[0791] Step 6:

[0792] The device will search for documents related to the specified appointment and automatically display them on the desktop. For example, if a meeting is scheduled for 10:00 a.m., the relevant documents will be displayed on the desktop 15 minutes before the meeting.

[0793] Step 7:

[0794] The device collects data from the camera and microphone built into the user's PC and analyzes it with an emotion engine, which evaluates the user's emotional state (stress, excitement, concentration, etc.) in real time.

[0795] Step 8:

[0796] The device monitors the user's work activity and if the work being performed differs from the scheduled data, a warning pop-up will be displayed after a specified time (e.g., 5 minutes). The content of the warning is adjusted by the emotion engine.

[0797] Step 9:

[0798] The device displays a pop-up warning: "You are currently working on something other than your scheduled schedule. Would you like to return to your scheduled schedule?" If the user's emotional state is stressed, the message will be displayed in a more gentle manner.

[0799] Step 10:

[0800] The server periodically analyzes schedule data and generates reminders when important events or deadlines are approaching, and the generated reminders are sent to the device.

[0801] Step 11:

[0802] Reminders received by the device are displayed as a pop-up on the desktop. The emotion engine adjusts the content and timing of the reminder according to the user's emotional state. For example, if the user is in a high-stress state, the reminder will be displayed at a more discreet time.

[0803] This series of processes enables users to carry out their work efficiently and also to receive support with optimal timing and expression based on their emotional state.

[0804] Example 2

[0805] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0806] Conventional schedule management systems often provide a function to retrieve schedule data and display necessary documents on the user's desktop to improve work efficiency. However, because they do not take the user's emotional state into consideration, excessive work can cause mental strain and reduce work efficiency. In addition, warnings and reminder notifications when the user is working outside of the schedule are displayed in a uniform manner, which makes it difficult to respond flexibly to individual situations.

[0807] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0808] In this invention, the server includes means for acquiring user schedule data, means for identifying necessary materials based on the acquired schedule data, means for automatically displaying the identified materials on the desktop of the user's terminal, means for recognizing the user's emotional state, and means for adjusting the display content of the desktop based on the recognized emotional state. This makes it possible to improve work efficiency and reduce mental burden by taking the user's emotional state into consideration.

[0809] "User schedule data" refers to information including the date, time, title, details, etc. of schedules that a user manages through an external calendar service or manual input.

[0810] The "means of acquisition" refers to a function by which the server collects user schedule data using a means such as an external calendar API.

[0811] The "means for identifying necessary materials" refers to an algorithm or program for identifying related materials or information based on the acquired schedule data.

[0812] The "means for automatic display" is a function for automatically displaying the identified material on the desktop of the user's terminal at the required timing.

[0813] The "means for recognizing the user's emotional state" is a system that uses an emotion analysis algorithm to evaluate the user's emotional state based on data acquired from the built-in camera and microphone.

[0814] The "means for adjusting desktop display content" is a function that dynamically changes the content and timing of information and notifications displayed on the desktop based on the recognized emotional state of the user.

[0815] The "means for monitoring activity" is a system that monitors active windows running on a terminal and user operations, and determines whether or not they are in accordance with a schedule.

[0816] The "means for displaying a warning" is a function that displays a notification or a pop-up to call the user's attention when the user is performing work outside of the schedule.

[0817] The "means for generating reminders" is a function for creating reminders to notify the user of important appointments and deadlines based on schedule data.

[0818] The "means for adjusting the content and timing of reminders" is a system that changes the content of reminder messages and the timing of notifications based on the user's emotional state.

[0819] This invention is a desktop magician system that improves users' work efficiency and mental well-being. It is mainly composed of a server and a user terminal, each of which performs specific functions to achieve performance optimization through schedule management and emotion recognition. Specific implementation methods are described below.

[0820] Basic system configuration

[0821] 1. Server Functions

[0822] Retrieving schedule data: The server retrieves the user's schedule data using the external calendar API. The retrieved data includes the date, time, title, and details of the event.

[0823] Data storage and management: The acquired schedule data is stored in a database on the server and kept accessible upon user request.

[0824] Reminder generation: The server analyzes the schedule data and generates reminders when important appointments or deadlines are approaching, and sends them to the user's terminal.

[0825] 2. Device Features

[0826] Data synchronization and storage: The terminal periodically receives schedule data sent from the server and stores it in local storage.

[0827] Automatic display of necessary materials: Based on the current time, schedule data and related materials are identified and displayed on the desktop. For example, if a meeting is scheduled, materials related to that meeting are automatically displayed.

[0828] Emotion Recognition: The emotion engine analyzes the user's emotional state in real time through the device's built-in camera and microphone. The emotion engine evaluates stress levels, concentration, and other factors and adjusts the content displayed on the desktop accordingly.

[0829] Activity monitoring and warnings: The device monitors the user's work activity and displays a warning pop-up if it detects any deviation from the scheduled activity. It also has the ability to adjust the warning content according to the user's emotional state.

[0830] Reminder notification: Notifies the user of reminders and adjusts the content and timing of notifications using an emotion engine.

[0831] Hardware and software used

[0832] Calendar API: Use an external calendar service (e.g., Google Calendar API) to retrieve schedule data.

[0833] Database: A database system (e.g., MySQL) for storing and managing schedule data on the server.

[0834] Emotion engine: Software that runs deep learning models to analyze a user's facial expressions and voice.

[0835] Built-in camera and microphone: Installed on the user device to collect data for emotion recognition.

[0836] Specific examples

[0837] 1. User's morning activities

[0838] The user starts up the PC and logs into Desktop Magician.

[0839] Server: Obtains the latest schedule data from an external calendar service and sends it to the device.

[0840] Device: Display materials related to the 10:00 AM meeting on your desktop.

[0841] 2. Monitoring during work

[0842] The user is browsing the web for a purpose other than "writing reports."

[0843] On your device: Display a warning popup saying "You are currently working on something other than your scheduled schedule. Would you like to return to your scheduled schedule?"

[0844] 3. Emotional engine response

[0845] When working for long periods of time, input from the camera and microphone is used to determine high stress levels.

[0846] Device: Display a notification that says "Take a 5-minute break to relax."

[0847] 4. Reminders

[0848] In the evening, a reminder appears on your device to let you know that a project is due tomorrow.

[0849] Server: Generates reminders and sends them to devices.

[0850] Device: If stress is low, notifications will be sent as normal, but if stress is high, notifications will be sent in a softer tone.

[0851] Prompt Sentence Examples

[0852] Prompt Sentence Examples

[0853] "Please explain in natural language the specific processing steps of the Desktop Magician system. Include the following items: obtaining user schedule data, automatically displaying materials, emotion recognition using an emotion engine, activity monitoring and alerts, and reminder notifications. Please clearly indicate the names and functions of the specific hardware and software used in all processing steps."

[0854] Specific questions

[0855] "After a user logs in, how does the server get the user's schedule data? And how is that data processed?"

[0856] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0857] Program processing flow and specific explanation

[0858] Step 1: User Login

[0859] User: Start the PC, click on the Desktop Magician app to go to the login screen, enter your username and password, and click the "Login" button.

[0860] Terminal: Sends the entered authentication information to the server.

[0861] Server: Compares the user information stored in the database, and if authentication is successful, generates an authentication token and sends it to the terminal.

[0862] Input: Username, Password

[0863] Output: Authentication token

[0864] Step 2: Retrieving and saving schedule data

[0865] Server: If authentication is successful, the external calendar API is used to retrieve the user's schedule data, including the event date, time, title, and details.

[0866] Server: Analyzes the acquired schedule data and stores it in a database.

[0867] Input: Schedule data from an external calendar API

[0868] Output: Schedule data stored in the server database

[0869] Step 3: Synchronize schedule data

[0870] Terminal: The Desktop Magician application running on the user's PC sends a schedule data update request to the server at regular intervals (e.g., every 5 minutes).

[0871] Server: Sends the latest schedule data to the terminal.

[0872] Terminal: Save the received schedule data in local storage.

[0873] Input: Schedule data update request from the server

[0874] Output: The latest schedule data saved in local storage

[0875] Step 4: Automatically display required materials

[0876] Device: Check your next appointment based on the current time. For example, if you have a meeting scheduled for 10:00 AM, identify the materials related to that meeting at 9:45 AM.

[0877] Device: The identified document will be displayed as a pop-up in a prominent location on your desktop.

[0878] Input: Schedule data in local storage

[0879] Output: Related documents displayed on your desktop

[0880] Step 5: Emotion Recognition

[0881] Device: Captures the user's facial expressions and voice in real time through the built-in camera and microphone.

[0882] Emotion engine: Analyzes acquired data using a deep learning model to evaluate the user's emotional state.

[0883] Device: Dynamically adjusts desktop display content and notification timing based on evaluation results.

[0884] Input: Data from the built-in camera and microphone

[0885] Output: Evaluated user emotional state and adjusted desktop display content

[0886] Step 6: Activity monitoring and alerts

[0887] Terminal: Monitors the title bar of the active window to see if the user is working according to a schedule.

[0888] Terminal: If the user is working outside of their schedule, a warning popup will appear after a specified time (e.g. 5 minutes).

[0889] Emotion engine: If the user's emotional state is high stress, the warning content will be softened.

[0890] Input: Active window title bar information

[0891] Output: Warning popup content

[0892] Step 7: Reminders

[0893] Server: Analyzes schedule data and generates reminders when important appointments or deadlines are approaching.

[0894] Server: Sends the generated reminders to the device.

[0895] Device: Notifies the user of received reminders and adjusts the content and timing using the emotion engine. For example, it sends a subtle notification such as "Remember the project submission deadline tomorrow."

[0896] Input: Schedule data

[0897] Output: Reminders notified to the user

[0898] (Application example 2)

[0899] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0900] In logistics center operations, managing schedules and related documents is essential for employees to work efficiently. However, doing this manually takes time and is prone to errors, which can lead to reduced work efficiency. Furthermore, long working hours and a stressful work environment can also put employees at risk of suffering from mental health issues, so a solution to these issues is needed.

[0901] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring user schedule data, means for identifying necessary materials based on the acquired schedule data, means for automatically displaying the identified materials on a display device on the user's terminal, means for recognizing the user's emotion, and means for adjusting the display content based on the recognized emotion. This allows employees to proceed with their work based on the automatically displayed schedule and related materials, and the use of emotion recognition technology enables them to work while maintaining their mental health.

[0902] "User schedule data" is information about the user's work schedule and task times.

[0903] "Means of acquisition" refers to the methods and systems used to collect schedule data from external databases and APIs.

[0904] "Necessary materials" refers to documents and information related to schedule data, such as work instructions and picking lists, which are necessary for carrying out work.

[0905] "Means for identifying" refers to a method or system that analyzes schedule data to determine what materials are needed at that time.

[0906] "User's Device" refers to a computer, smartphone, smart glasses, or other electronic device used by a User to conduct business.

[0907] A "display device" is a screen or display that allows a user to visually view information.

[0908] "Means for automatic display" is a mechanism for automatically displaying necessary materials on the display device of a terminal under certain conditions.

[0909] "Means for recognizing emotions" refers to technologies or systems that analyze the user's facial expressions and vocal characteristics to infer their emotional state at that time.

[0910] The "means for adjusting the displayed content" refers to a method or system for changing the type of information displayed or the way it is expressed based on the recognized emotion.

[0911] The "activity monitoring means" is a system for observing and recording the user's work status in real time.

[0912] The "means for displaying warnings" is a mechanism for displaying warning messages on the terminal display when the user's current activity deviates from the schedule or when a problem occurs.

[0913] The "means for generating reminders" is a notification function that notifies the user of important appointments and task deadlines based on schedule data.

[0914] The "means for displaying a reminder" is a system for displaying the generated reminder on the display device of the user's terminal.

[0915] "Means for adjusting reminder content and timing" refers to a system or method for changing the content of reminder messages and / or notification timing in consideration of a perceived emotional state.

[0916] This invention provides a "Logistics Magician" system for improving the work efficiency and mental health of employees at logistics centers. This system acquires the user's schedule data and automatically presents relevant materials based on that data, thereby improving work efficiency. It also uses emotion recognition technology to support the mental health of employees.

[0917] Basic system configuration

[0918] The basic configuration of the system is as follows:

[0919] 1. Server:

[0920] Obtaining schedule data: The server uses the API to obtain the user's schedule data from an external database.

[0921] Data management and storage: The acquired schedule data is stored in a database and the necessary data is sent to the terminal.

[0922] Generate reminders: Analyze schedule data and generate reminders.

[0923] 2. Terminal:

[0924] Schedule data synchronization: The terminal receives schedule data provided by the server and stores it in local storage.

[0925] Automatic display of materials: Based on the current time, schedule data and related materials are identified and displayed on the display device.

[0926] Activity monitoring and alerts: Monitors users' work activity and displays alerts if it deviates from the schedule.

[0927] Emotion engine: Using emotion recognition technology, it analyzes the user's emotional state and adjusts the display content and reminder content.

[0928] Specific processing explanation

[0929] 1. Data acquisition and display:

[0930] Hardware: Smart glasses, smartphones

[0931] The server retrieves the user's schedule data from an external database or API and stores it in the database. The device receives the latest schedule data from the server at regular intervals and stores it in local storage. When executed, the schedule data and related materials are displayed on the display device based on the current time.

[0932] Example: If a user has a meeting scheduled at 10:00 AM, materials related to that meeting will automatically appear on the display at 9:45 AM.

[0933] 2. Emotion recognition:

[0934] Software: OpenCV

[0935] The system analyzes the device's camera footage in real time and estimates the user's emotional state from their facial expressions, thereby assessing their stress level and fatigue.

[0936] Example: If the user feels fatigued, the device will notify them to "take a 5-minute break to relax."

[0937] 3. Notifications and Reminders:

[0938] It notifies users of important tasks and break times based on schedule data, and adjusts the content and timing of reminder messages based on emotional state.

[0939] Example: When a user is in a high-stress state, a reminder is displayed discreetly: "It's time for your next task. Please be mindful."

[0940] Prompt Sentence Examples

[0941] "If the user is not performing the current task accurately, display 'Are you continuing to work as planned?' and generate a prompt sentence recommending a break based on the emotion recognition results."

[0942] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0943] Step 1:

[0944] Acquiring and managing schedule data

[0945] Server: When a user logs in, the server uses an API to retrieve the user's schedule data from an external database, including the date, time, title, and details of the event.

[0946] Input: User ID, API authentication information.

[0947] Data processing: Call the API and receive schedule data in JSON format as a response.

[0948] Output: Schedule data in JSON format.

[0949] What happens: The server runs this process periodically and saves the schedule data to a database to keep it up to date.

[0950] Step 2:

[0951] Schedule data synchronization

[0952] Terminal: The terminal receives schedule data provided by the server at regular intervals and stores it in local storage.

[0953] Input: The request to the server.

[0954] Data calculation: Analyzes the JSON data sent from the server and saves it in local storage.

[0955] Output: The latest schedule data is saved locally.

[0956] Specific operation: The device sends a request to the server every five minutes to obtain the latest schedule data and update it to local storage.

[0957] Step 3:

[0958] Automatic display of materials

[0959] Terminal: Based on the current time, schedule data and related materials are identified and displayed on the display device.

[0960] Input: Local storage schedule data, current time.

[0961] Data calculation: Compare the current time with schedule data to identify the corresponding materials.

[0962] Output: Relevant material displayed on a display device.

[0963] What it does: If you have a meeting scheduled for 10:00 AM, presentations and documents related to that meeting will automatically appear at 9:45 AM.

[0964] Step 4:

[0965] emotion recognition

[0966] Device: Analyzes camera images from smart glasses or smartphones in real time and estimates the user's emotional state from their facial expressions.

[0967] Input: Camera footage.

[0968] Data calculation: Using OpenCV, emotions are estimated using a facial expression analysis model.

[0969] Output: Emotional state, such as stress level and fatigue level.

[0970] What it does: The device's camera captures the user's face and analyzes the image in real time to determine whether the user is tired.

[0971] Step 5:

[0972] Generate and coordinate notifications and reminders

[0973] Server (generation) and device (adjustment): The server generates reminders based on schedule data and sends them to the device. The device adjusts the content and timing of the reminders based on the recognized emotions.

[0974] Input: Schedule data, emotional state.

[0975] Data calculation: Analyze schedule data and adjust reminders based on emotional state.

[0976] Output: The adjusted reminder is displayed on the display device.

[0977] What it does: If the user is in a high-stress state, the reminder is subtly displayed as "It's time for your next task. Please be mindful."

[0978] Step 6:

[0979] Activity monitoring and alerts

[0980] Terminal: Monitors the user's work activity and displays a warning if it differs from the scheduled data.

[0981] Input: Current activity, schedule data.

[0982] Data calculation: Compares activity content with schedule data and generates a warning if there is a discrepancy.

[0983] Output: A warning message is displayed.

[0984] What it does: If a user is browsing a website other than the scheduled task, a message will pop up asking, "Are you still working as planned?"

[0985] Prompt Sentence Examples

[0986] "If the user is not performing the current task accurately, display 'Are you continuing to work as planned?' and generate a prompt sentence recommending a break based on the emotion recognition results."

[0987] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0988] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0989] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0990] [Third embodiment]

[0991] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0992] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0993] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0994] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0995] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0996] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0997] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0998] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0999] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1000] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1001] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1002] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1003] This invention is a desktop magician system for improving the user's work efficiency. It aims to acquire the user's schedule data and automatically display the necessary documents on the desktop based on that data. The system is realized by linking a server and a terminal.

[1004] System configuration

[1005] The basic system configuration is as follows:

[1006] 1. Server:

[1007] Retrieve user schedule data from external services (e.g., Google Calendar or Outlook Calendar).

[1008] The acquired schedule data is managed and saved, and the necessary data is sent to the user terminal.

[1009] Analyze schedule data and generate reminders.

[1010] 2. Terminal:

[1011] Schedule data provided by the server is received and stored in local storage.

[1012] Display the necessary materials on your desktop based on the current time.

[1013] Monitor users' work activity and display alerts when work is outside of schedule.

[1014] Notify users of reminders.

[1015] Overview of program processing

[1016] 1. Acquiring and saving schedule data

[1017] Server: When a user logs in, the server uses an API to retrieve the user's schedule data and saves it in a database. This data includes the date, time, title, and details of the event.

[1018] 2. Schedule data synchronization

[1019] Server: Updates the schedule data at regular intervals to keep it up to date.

[1020] Terminal: Receives schedule data updates from the server at regular intervals and saves them in local storage.

[1021] 3. Automatic display of materials

[1022] Terminal: Based on the current time, schedule data and related materials are identified and displayed on the desktop. For example, if a meeting is scheduled for 10:00 AM, materials related to that meeting will be displayed on the desktop 15 minutes before the meeting.

[1023] 4. Activity Monitoring and Alerts

[1024] Terminal: Monitors user activity and displays a warning pop-up after a certain period of time if the user is performing tasks that differ from the scheduled data. This pop-up prompts the user to recheck their schedule.

[1025] 5. Reminders

[1026] Server: Analyzes schedule data and generates reminders when important events or deadlines are approaching. Reminders are generated at set times (e.g., 3 days before, 1 day before).

[1027] Device: Reminder notifications are received and displayed on the desktop. After the user confirms the reminder, they can also use the snooze function to be notified again.

[1028] Specific examples

[1029] User's morning activities:

[1030] 1. A user starts up their PC in the morning and logs into Desktop Magician.

[1031] 2. Server: Obtains the user's latest schedule data from an external calendar service and sends it to the device.

[1032] 3. Terminal: Based on the received data, materials related to the meeting at 10:00 AM today are displayed on the desktop.

[1033] Monitoring during work:

[1034] 1. In the morning, the user is browsing the web for an activity other than the scheduled "report writing."

[1035] 2. Terminal: Monitor the title bar of the active window and display a warning popup if it is different from the schedule.

[1036] 3. A pop-up will appear saying, "You are currently working on something different from your schedule. Would you like to return to your schedule?" and the user will select "Yes."

[1037] Reminder:

[1038] 1. In the evening, a reminder appears on your device to let you know that a project is due tomorrow.

[1039] 2. Server: Generates reminders the day before and sends them to the device.

[1040] 3. Device: Reminders pop up on the desktop so users can take action.

[1041] As described above, the present invention provides an environment in which a user can work efficiently through a specific program and its processing.

[1042] The processing flow will be explained below.

[1043] Step 1:

[1044] The user logs in to Desktop Magician. Once logged in, the system sends the user authentication information to the server.

[1045] Step 2:

[1046] The server receives the user's authentication information and verifies that the user is a valid user. If authentication is successful, the server retrieves the user's schedule data (e.g., Google Calendar or Outlook Calendar) from an external API.

[1047] Step 3:

[1048] The server analyzes the acquired schedule data, extracts the user's schedule items, and stores them in a database, including the schedule date and time, title, details, and links to attachments.

[1049] Step 4:

[1050] The device requests synchronization of schedule data from the server at regular intervals (for example, every morning), and the server sends the latest schedule data to the device.

[1051] Step 5:

[1052] The device stores the received schedule data in local storage, then compares the current time with the schedule data to identify events scheduled within the next few hours.

[1053] Step 6:

[1054] The device will search for materials related to the identified event and automatically display them on the desktop. For example, if a meeting is scheduled for 10:00 a.m., materials related to this meeting will be displayed on the desktop 15 minutes before the meeting.

[1055] Step 7:

[1056] The device monitors the user's work activity and displays a warning popup after a specified time (e.g., 5 minutes) if the user is not performing the scheduled task.

[1057] Step 8:

[1058] The device will display a pop-up warning to the user, saying, "You are currently working on something different from your scheduled schedule. Would you like to return to your scheduled schedule?" If the user selects "Yes," they will be provided with a link to easily return to their scheduled materials or applications.

[1059] Step 9:

[1060] The server periodically analyzes schedule data and generates reminders when important events or deadlines are approaching, which are then sent to the device.

[1061] Step 10:

[1062] The device will display the received reminder as a pop-up on the desktop, and after the user confirms the reminder, they can set the snooze function to be notified again later.

[1063] This series of processes allows users to concentrate on their work efficiently and makes schedule management easier.

[1064] Example 1

[1065] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1066] Conventional schedule management systems require users to prepare materials themselves, and do not provide sufficient warnings or reminders when they are working outside of the schedule, which can lead to reduced work efficiency.

[1067] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1068] In this invention, the server includes means for acquiring user schedule data from an external service, means for saving the acquired schedule data in a database, and means for periodically updating the schedule data to keep it up to date. This allows necessary materials to be automatically displayed on the desktop based on the latest schedule data, improving the user's work efficiency.

[1069] The terminal also includes means for identifying materials relevant to the current time based on the schedule data and automatically displaying them on the desktop, means for monitoring the user's activity during work and displaying a warning if the user is performing work that differs from the schedule, means for analyzing the schedule data and generating a reminder when an important appointment or deadline approaches, and means for displaying the generated reminder on the desktop. This enables the user to accurately perform the scheduled work and proceed with work without missing important appointments or deadlines.

[1070] "User schedule data" refers to information about plans and events that a user has registered in an external service (for example, a calendar service), and includes information such as date and time, title, and details.

[1071] "External service" refers to an online platform for managing, storing, and providing users' schedule data, such as a calendar application or a cloud-based schedule management service.

[1072] A "database" is a system that organizes, stores, and manages acquired schedule data.

[1073] "Regularly updated" means that schedule data is reacquired from an external service at regular intervals to keep the contents of the database up to date.

[1074] "Necessary materials" refers to documents and files related to scheduled tasks and events based on the user's schedule data.

[1075] "Automatically displaying on the desktop" means displaying materials on the screen of the user's terminal without the need for manual operation.

[1076] "Terminal" refers to an electronic computer that allows a user to use the Desktop Magician System, and includes desktop PCs, notebook PCs, tablets, etc.

[1077] "Activity" refers to the work or operation that a user performs on a terminal, and indicates the type of work that can be determined, for example, by the title bar of an active window.

[1078] A "warning popup" is a small window that displays a warning or instructions when the user is performing work that is different from the schedule.

[1079] A "reminder" is a message that notifies the user when an important event or deadline is approaching, and is generated at a specified timing.

[1080] "Local storage" refers to a storage device (such as a hard disk or SSD) built into the user's terminal, and is used to store schedule data and related materials received from the server.

[1081] This invention is a desktop magician system for improving the user's work efficiency. The system acquires the user's schedule data and automatically displays the necessary documents on the desktop based on that data. The system is realized by linking a server and a terminal.

[1082] server

[1083] When a user logs in to the Desktop Magician system, the server uses an API to retrieve the user's schedule data from external services (such as Google Calendar or Outlook Calendar). This retrieved data includes the date, time, title, and details of the appointment. The retrieved data is then stored in a database on the server. The server then updates the schedule data at regular intervals to keep it up to date. In addition, the server analyzes the schedule data and generates reminders when important appointments or deadlines are approaching, which are then sent to the device.

[1084] Terminal

[1085] The device periodically receives schedule data provided by the server and saves it in local storage. Based on the current time, it identifies the schedule data and related materials and automatically displays them on the desktop. The device also monitors the user's work activity and displays a warning pop-up after a certain period of time if the user is performing work that differs from the schedule data. This pop-up prompts the user to reconfirm their schedule. It also sends reminder notifications, providing information for the user to take action. Reminder notifications are displayed as a pop-up on the desktop, and after the user confirms the reminder, they can use the snooze function to be notified again.

[1086] Usage example

[1087] User's morning activities:

[1088] A user starts up their PC in the morning and logs into Desktop Magician.

[1089] The server obtains the user's latest schedule data from an external calendar service and sends it to the terminal.

[1090] Based on the data received by the device, materials related to the meeting at 10:00 a.m. today will be displayed on the desktop.

[1091] Monitoring during work:

[1092] In the morning, the user is browsing the web, which is different from the scheduled "report writing" activity.

[1093] The terminal monitors the title bar of the active window and displays a warning popup if it is different from what is expected.

[1094] The popup displays "You are currently working on something different from your schedule. Would you like to return to your schedule?" and the user selects "Yes."

[1095] Reminder:

[1096] In the evening, a reminder appears on your device to let you know that a project is due tomorrow.

[1097] The server generates a reminder the day before and sends it to the device.

[1098] The device will pop up a reminder on your desktop so you can take action.

[1099] Example of input prompt for generative AI model

[1100] "A user logs into the Desktop Magician system. I want the documents related to the 10 AM meeting to be displayed on the desktop. I also want the system to monitor the user's work activity and warn them if they are working in a different way."

[1101] As described above, the present invention effectively manages a user's schedule through cooperation between a server and a terminal, and provides necessary materials at the appropriate time, thereby improving the user's work efficiency.

[1102] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1103] Step 1: User Login

[1104] Input: User ID and password

[1105] Process: The user starts up the PC and logs in to the Desktop Magician system. The entered user ID and password are sent to the server for authentication.

[1106] Data processing: The server authenticates the user ID and password and generates a success or failure result.

[1107] Output: If authentication is successful, obtain the user's basic information and authorization information and send it to the terminal. If authentication is unsuccessful, display an error message.

[1108] Specific operation: The user enters the user ID and password on the login screen and clicks the login button. The server performs authentication, and if successful, the main screen is displayed.

[1109] Step 2: Retrieving and saving schedule data

[1110] Input: User login information

[1111] Processing: Based on the user's login information, the server uses an API to retrieve schedule data from an external calendar service (e.g., Google Calendar, Outlook Calendar).

[1112] Data processing: The acquired schedule data includes the appointment date, time, title, and details, and is saved in a database on the server.

[1113] Output: Schedule data stored in the database

[1114] What happens: The server sends an API request using the user's calendar account information to retrieve and store schedule data.

[1115] Step 3: Synchronize schedule data

[1116] Input: API request to external calendar service

[1117] Processing: The server periodically sends an API request to the external calendar service to retrieve the latest schedule data.

[1118] Data processing: Compare new or changed schedule data with the database and update it to the latest version.

[1119] Output: Updated database schedule data

[1120] Specific operation: The server makes an API request every hour or at a set interval and updates the database with the retrieved data.

[1121] Step 4: Send schedule data

[1122] Input: Schedule data from the database

[1123] Processing: The terminal periodically sends a request to the server and receives the latest schedule data.

[1124] Data processing: Save the received schedule data to local storage.

[1125] Output: Schedule data in local storage

[1126] Specific operation: The device requests schedule data from the server every 30 minutes and saves the received data.

[1127] Step 5: Automatically display materials

[1128] Input: Local storage schedule data and current time

[1129] Processing: Based on the schedule data, the terminal identifies materials relevant to the current time and displays them on the desktop.

[1130] Data processing: Analyze schedule data and obtain paths to identified materials.

[1131] Output: Documents displayed on the desktop

[1132] Specific operation: The device checks the current time, and if there is a meeting at 10:00, the meeting materials will be displayed on the desktop at 9:45.

[1133] Step 6: Activity monitoring and alerts

[1134] Input: User activity data (active window title bar)

[1135] Action: The device monitors the user's activity and displays a warning pop-up if the user is performing an activity that differs from the scheduled data.

[1136] Data processing: Analyze activity data and compare it with planned work.

[1137] Output: Warning popup

[1138] Specific behavior: If the user is browsing the web in a way that is not part of their schedule, the message "You are currently working on something different from your schedule. Would you like to return to your schedule?" will be displayed.

[1139] Step 7: Reminders

[1140] Input: Schedule data from the database

[1141] Processing: The server analyzes the schedule data and generates reminders when important events or deadlines approach.

[1142] Data processing: Identify important events and deadlines and generate reminder messages.

[1143] Output: The generated reminder is sent to the device and displayed on the desktop.

[1144] What happens: The server generates a reminder for the previous day, and the device displays the reminder on the desktop.

[1145] Example of input prompt for generative AI model

[1146] "A user logs into the Desktop Magician system. I want the documents related to the 10 AM meeting to be displayed on the desktop. I also want the system to monitor the user's work activity and warn them if they are working in a different way."

[1147] (Application example 1)

[1148] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1149] Conventional factory robots did not properly manage work schedules or monitor work status in real time, making it difficult to achieve efficient production activities. Also, when workers performed tasks that deviated from the schedule, appropriate measures could not be taken. Furthermore, the lack of reminders for important tasks and deadlines meant that efficient work operations could not be achieved.

[1150] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1151] In this invention, the server includes a means for acquiring a user's schedule data, a means for identifying necessary documents based on the acquired schedule data, and a means for automatically displaying the identified documents on the desktop of the user's terminal. This enables effective schedule management and real-time monitoring of work status, which was difficult with conventional factory robots. It also enables warnings when work is being performed that deviates from the schedule and reminders for important work and deadlines, thereby achieving efficient work management.

[1152] "User schedule data" refers to information relating to the date, time, schedule contents, etc. that the user has planned in advance.

[1153] "Means of acquisition" refers to application programs or APIs for acquiring user schedule data from external calendar services, etc.

[1154] "Required materials" refers to documents and digital files that are required to perform a specific task based on the schedule data.

[1155] "Means for identification" refers to algorithms or software that analyze acquired schedule data and identify necessary materials based on that data.

[1156] "Means for automatic display" refers to a program or tool for automatically displaying the identified material on the user's terminal.

[1157] A "factory robot" refers to an automated mechanical device designed to perform production tasks in a factory.

[1158] "Current activity" refers to a specific task or activity that a user or factory robot is currently performing.

[1159] "Monitoring means" refers to programs and sensors that observe and record the work activities of users or factory robots in real time.

[1160] The "means for displaying a warning" refers to a program or device for displaying a warning message when a user or a factory robot is performing an activity that differs from the schedule.

[1161] A "reminder" is a message that notifies a user when an important appointment or deadline is approaching, based on the user's schedule data.

[1162] "Means for generating" refers to programs or software for generating reminders based on schedule data.

[1163] "Means for monitoring progress" refers to programs and sensors that monitor the progress of tasks being performed by factory robots in real time.

[1164] "Efficient work instructions" refers to specific instructions and guidance provided to factory robots to help them work more efficiently.

[1165] This invention is a system for improving the work efficiency of factory robots. This system acquires user schedule data, prepares necessary materials and tools based on that data, and provides them to the factory robots at the appropriate time, thereby improving work efficiency.

[1166] System configuration

[1167] The basic system configuration is as follows:

[1168] 1. Server:

[1169] Obtain user schedule data from an external service (e.g., a calendar service).

[1170] The acquired schedule data is managed and saved, and the necessary data is sent to the user's device or factory robot.

[1171] Analyze schedule data and generate reminders.

[1172] 2. Terminal:

[1173] Schedule data provided by the server is received and stored in local storage.

[1174] Display required materials based on the current time.

[1175] 3. Factory robots:

[1176] Automatically prepare materials and tools.

[1177] Monitor work activity and display alerts when schedules deviate.

[1178] Receive reminders and provide work instructions.

[1179] Monitor progress and provide efficient work instructions.

[1180] Overview of program processing

[1181] 1. Acquiring and saving schedule data

[1182] Server: When a user logs in, the server uses an API to retrieve the user's schedule data and saves it in a database. This data includes the date, time, and content of the schedule.

[1183] 2. Schedule data synchronization

[1184] Server: Updates the schedule data at regular intervals to keep it up to date.

[1185] Terminal: Receives schedule data updates from the server at regular intervals and saves them in local storage.

[1186] 3. Automatic display of materials

[1187] Terminal: Based on the current time, schedule data and related materials are identified and displayed on the desktop. For example, if a welding job is scheduled for 10:00 AM, materials related to that job will be displayed on the desktop 15 minutes beforehand.

[1188] 4. Activity Monitoring and Alerts

[1189] Factory robots: Monitor activity and display a warning pop-up after a certain time if the robot is performing work that is different from the schedule, prompting the worker to reconfirm the schedule.

[1190] 5. Reminders

[1191] Server: Analyzes schedule data and generates reminders when important events or deadlines are approaching. Reminders are generated at a set time (e.g., one day before).

[1192] Factory robots: receive reminder notifications and provide work instructions.

[1193] Specific examples

[1194] Consider a scenario in which a specific factory robot is scheduled to perform welding work starting at 10:00 AM. When a user logs into the system in the morning, the server retrieves the user's schedule data from an external calendar service and sends it to the terminal and factory robot. The terminal displays materials related to the 10:00 AM welding work on its desktop, and the factory robot prepares the tools necessary for welding. If the robot is performing a different activity during the work, a warning is displayed and the user is asked to confirm. Furthermore, the server notifies the user that the work deadline is one day away, generates a reminder, and provides instructions to the factory robot.

[1195] Prompt Sentence Examples

[1196] To check the user's morning schedule, retrieve the factory robot's schedule data and display the materials related to the 10:00 a.m. work. If there is any work outside the schedule, display a warning and generate a reminder within 24 hours.

[1197] In this way, the present invention improves the work efficiency of factory robots through specific programs and means, and realizes effective schedule management and real-time work monitoring.

[1198] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1199] Step 1:

[1200] When a user logs in, the server accesses the API of the external calendar service to obtain the user's schedule data. Specifically, it uses the user ID as an API parameter and sends a request. Based on this input, the schedule data (schedule date and time, title, and details) is obtained in JSON format. The obtained data is saved in the server's database.

[1201] Step 2:

[1202] The server updates the schedule data at regular intervals. To update, it accesses the API of the external calendar service again to obtain the latest schedule data. If new data exists, it updates the database contents. This input data keeps the server database up to date with the latest schedule data.

[1203] Step 3:

[1204] The terminal periodically obtains schedule data updates from the server. The terminal receives the latest schedule data from the server and stores the data in the terminal's local storage. This input data allows the terminal to access the latest schedule data.

[1205] Step 4:

[1206] The terminal references the schedule data based on the current time and identifies the relevant materials 15 minutes before the next scheduled appointment. For example, if welding work is scheduled for 10:00 a.m., the materials needed for that work will be identified and displayed on the desktop. Based on this input, the file path of the identified materials is output, and the materials are displayed on the desktop.

[1207] Step 5:

[1208] The factory robot automatically prepares the identified materials and tools. Based on the schedule data, it generates a list of tools required for the day's work and prepares them. For example, in the case of welding work, welding machines and protective equipment are prepared. This input data allows for efficient work preparation.

[1209] Step 6:

[1210] Factory robots use sensors and programs to monitor their work activities. They observe the current work status in real time and compare it with the schedule data. If the work differs from the schedule, a warning pop-up is displayed. This input data detects discrepancies and outputs a warning to the user or robot.

[1211] Step 7:

[1212] The server analyzes the schedule data and generates reminders when important appointments or deadlines are approaching. These reminders are generated at a set time (e.g., one day before) based on the schedule date and time. The generated reminders are sent to the user's device and factory robots. This input data allows the reminders to be output at the appropriate time.

[1213] Step 8:

[1214] The factory robot receives the reminder notification and provides the user with work instructions. For example, a notification is received the day before welding work and an instruction is issued to start preparations the next day. Based on this input data, the reminder content is output and the factory robot is notified reliably.

[1215] This enables effective schedule management, real-time monitoring of work status, and efficient provision of work instructions, which were difficult with conventional factory robots.

[1216] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1217] This invention is a desktop magician system for improving the user's work efficiency and mental well-being. It acquires the user's schedule data and automatically displays necessary documents on the desktop based on that data. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized support.

[1218] System configuration

[1219] The basic system configuration is as follows:

[1220] 1. Server:

[1221] Obtaining user schedule data from an external service (e.g., a calendar API).

[1222] The acquired schedule data is managed and saved, and the necessary data is sent to the user terminal.

[1223] Analyze schedule data and generate reminders.

[1224] 2. Terminal:

[1225] Schedule data provided by the server is received and stored in local storage.

[1226] Display the necessary materials on your desktop based on the current time.

[1227] Monitor users' work activity and display alerts when work is outside of schedule.

[1228] Notify users of reminders.

[1229] It has an emotion engine that recognizes the user's emotions and adjusts desktop displays, alerts, and reminder notifications based on the user's emotional state.

[1230] Overview of program processing

[1231] 1. Acquiring and saving schedule data

[1232] Server: When a user logs in, the server uses an API to retrieve the user's schedule data and saves it in a database. This data includes the date, time, title, and details of the event.

[1233] 2. Schedule data synchronization

[1234] Server: Updates the schedule data at regular intervals to keep it up to date.

[1235] Terminal: Receives schedule data updates from the server at regular intervals and saves them in local storage.

[1236] 3. Automatic display of materials

[1237] Terminal: Based on the current time, schedule data and related materials are identified and displayed on the desktop. For example, if a meeting is scheduled for 10:00 AM, materials related to that meeting will be displayed on the desktop 15 minutes before the meeting.

[1238] 4. Emotion Recognition by Emotion Engine

[1239] Device: While the user is using the PC, the emotion engine processes input data from the built-in camera and microphone and analyzes the user's emotional state in real time. The emotional state is expressed as parameters such as stress level, excitement level, and concentration.

[1240] 5. Activity Monitoring and Alerts

[1241] Terminal: Monitors user activity and displays a warning pop-up after a specified time (e.g., 5 minutes) if the user is performing tasks that differ from the scheduled data. If the user is feeling overly stressed, the emotion engine can soften or postpone the warning.

[1242] 6. Reminders

[1243] Server: Analyzes schedule data and generates reminders when important events or deadlines are approaching. The generated reminders are sent to the device.

[1244] On-device: The emotional engine takes into account the user's emotional state and adjusts the content and timing of reminders. For example, if the user is feeling stressed, the reminder will be more toned down.

[1245] Specific examples

[1246] User's morning activities:

[1247] 1. A user starts up their PC in the morning and logs into Desktop Magician.

[1248] 2. Server: Obtains the user's latest schedule data from an external calendar service and sends it to the device.

[1249] 3. Terminal: Based on the received data, materials related to the meeting at 10:00 AM today are displayed on the desktop.

[1250] Monitoring during work:

[1251] 1. In the morning, the user is browsing the web for an activity other than the scheduled "report writing."

[1252] 2. Terminal: Monitor the title bar of the active window and display a warning popup if it is different from the schedule.

[1253] 3. The popup will say "You are currently working on something different from your schedule. Would you like to return to your schedule?" If the user is feeling annoyed, the warning will be more subtle.

[1254] Emotion engine response:

[1255] 1. When a user continues working for a long time, input from the camera and microphone indicates that stress is high.

[1256] 2. Device: Based on the emotion engine, notify "Take a 5-minute break to relax."

[1257] Reminder:

[1258] 1. In the evening, a reminder appears on your device to let you know that a project is due tomorrow.

[1259] 2. Server: Generates reminders the day before and sends them to the device.

[1260] 3. Device: If the user's stress level is low, the emotion engine will notify them normally, but if it is high, it will notify them in a softer way.

[1261] As described above, the present invention enables users to perform their work efficiently and in a mentally healthy environment through a specific program and its processing that takes into account the user's emotional state.

[1262] The processing flow will be explained below.

[1263] Step 1:

[1264] A user logs in to Desktop Magician. Upon login, the system sends the user's authentication information to the server.

[1265] Step 2:

[1266] The server receives the user's authentication information and verifies that the user is a valid user. If authentication is successful, the server retrieves the user's schedule data from an external API (e.g., a calendar API) and saves it in a database.

[1267] Step 3:

[1268] The server analyzes the schedule data and extracts the user's schedule items (such as the schedule date and time, title, details, and attachment links).

[1269] Step 4:

[1270] The device requests synchronization of schedule data from the server at regular intervals (for example, every morning), and the server sends the latest schedule data to the device.

[1271] Step 5:

[1272] The device saves the received schedule data in local storage. After saving, it compares the current time with the schedule data to identify upcoming events.

[1273] Step 6:

[1274] The device will search for documents related to the specified appointment and automatically display them on the desktop. For example, if a meeting is scheduled for 10:00 a.m., the relevant documents will be displayed on the desktop 15 minutes before the meeting.

[1275] Step 7:

[1276] The device collects data from the camera and microphone built into the user's PC and analyzes it with an emotion engine, which evaluates the user's emotional state (stress, excitement, concentration, etc.) in real time.

[1277] Step 8:

[1278] The device monitors the user's work activity and if the work being performed differs from the scheduled data, a warning pop-up will be displayed after a specified time (e.g., 5 minutes). The content of the warning is adjusted by the emotion engine.

[1279] Step 9:

[1280] The device displays a pop-up warning: "You are currently working on something other than your scheduled schedule. Would you like to return to your scheduled schedule?" If the user's emotional state is stressed, the message will be displayed in a more gentle manner.

[1281] Step 10:

[1282] The server periodically analyzes schedule data and generates reminders when important events or deadlines are approaching, and the generated reminders are sent to the device.

[1283] Step 11:

[1284] Reminders received by the device are displayed as a pop-up on the desktop. The emotion engine adjusts the content and timing of the reminder according to the user's emotional state. For example, if the user is in a high-stress state, the reminder will be displayed at a more discreet time.

[1285] This series of processes enables users to carry out their work efficiently and also to receive support with optimal timing and expression based on their emotional state.

[1286] Example 2

[1287] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1288] Conventional schedule management systems often provide a function to retrieve schedule data and display necessary documents on the user's desktop to improve work efficiency. However, because they do not take the user's emotional state into consideration, excessive work can cause mental strain and reduce work efficiency. In addition, warnings and reminder notifications when the user is working outside of the schedule are displayed in a uniform manner, which makes it difficult to respond flexibly to individual situations.

[1289] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1290] In this invention, the server includes means for acquiring user schedule data, means for identifying necessary materials based on the acquired schedule data, means for automatically displaying the identified materials on the desktop of the user's terminal, means for recognizing the user's emotional state, and means for adjusting the display content of the desktop based on the recognized emotional state. This makes it possible to improve work efficiency and reduce mental burden by taking the user's emotional state into consideration.

[1291] "User schedule data" refers to information including the date, time, title, details, etc. of schedules that a user manages through an external calendar service or manual input.

[1292] The "means of acquisition" refers to a function by which the server collects user schedule data using a means such as an external calendar API.

[1293] The "means for identifying necessary materials" refers to an algorithm or program for identifying related materials or information based on the acquired schedule data.

[1294] The "means for automatic display" is a function for automatically displaying the identified material on the desktop of the user's terminal at the required timing.

[1295] The "means for recognizing the user's emotional state" is a system that uses an emotion analysis algorithm to evaluate the user's emotional state based on data acquired from the built-in camera and microphone.

[1296] The "means for adjusting desktop display content" is a function that dynamically changes the content and timing of information and notifications displayed on the desktop based on the recognized emotional state of the user.

[1297] The "means for monitoring activity" is a system that monitors active windows running on a terminal and user operations, and determines whether or not they are in accordance with a schedule.

[1298] The "means for displaying a warning" is a function that displays a notification or a pop-up to call the user's attention when the user is performing work outside of the schedule.

[1299] The "means for generating reminders" is a function for creating reminders to notify the user of important appointments and deadlines based on schedule data.

[1300] The "means for adjusting the content and timing of reminders" is a system that changes the content of reminder messages and the timing of notifications based on the user's emotional state.

[1301] This invention is a desktop magician system that improves users' work efficiency and mental well-being. It is mainly composed of a server and a user terminal, each of which performs specific functions to achieve performance optimization through schedule management and emotion recognition. Specific implementation methods are described below.

[1302] Basic system configuration

[1303] 1. Server Functions

[1304] Retrieving schedule data: The server retrieves the user's schedule data using the external calendar API. The retrieved data includes the date, time, title, and details of the event.

[1305] Data storage and management: The acquired schedule data is stored in a database on the server and kept accessible upon user request.

[1306] Reminder generation: The server analyzes the schedule data and generates reminders when important appointments or deadlines are approaching, and sends them to the user's terminal.

[1307] 2. Device Features

[1308] Data synchronization and storage: The terminal periodically receives schedule data sent from the server and stores it in local storage.

[1309] Automatic display of necessary materials: Based on the current time, schedule data and related materials are identified and displayed on the desktop. For example, if a meeting is scheduled, materials related to that meeting are automatically displayed.

[1310] Emotion Recognition: The emotion engine analyzes the user's emotional state in real time through the device's built-in camera and microphone. The emotion engine evaluates stress levels, concentration, and other factors and adjusts the content displayed on the desktop accordingly.

[1311] Activity monitoring and warnings: The device monitors the user's work activity and displays a warning pop-up if it detects any deviation from the scheduled activity. It also has the ability to adjust the warning content according to the user's emotional state.

[1312] Reminder notification: Notifies the user of reminders and adjusts the content and timing of notifications using an emotion engine.

[1313] Hardware and software used

[1314] Calendar API: Use an external calendar service (e.g., Google Calendar API) to retrieve schedule data.

[1315] Database: A database system (e.g., MySQL) for storing and managing schedule data on the server.

[1316] Emotion engine: Software that runs deep learning models to analyze a user's facial expressions and voice.

[1317] Built-in camera and microphone: Installed on the user device to collect data for emotion recognition.

[1318] Specific examples

[1319] 1. User's morning activities

[1320] The user starts up the PC and logs into Desktop Magician.

[1321] Server: Obtains the latest schedule data from an external calendar service and sends it to the device.

[1322] Device: Display materials related to the 10:00 AM meeting on your desktop.

[1323] 2. Monitoring during work

[1324] The user is browsing the web for a purpose other than "writing reports."

[1325] On your device: Display a warning popup saying "You are currently working on something other than your scheduled schedule. Would you like to return to your scheduled schedule?"

[1326] 3. Emotional engine response

[1327] When working for long periods of time, input from the camera and microphone is used to determine high stress levels.

[1328] Device: Display a notification that says "Take a 5-minute break to relax."

[1329] 4. Reminders

[1330] In the evening, a reminder appears on your device to let you know that a project is due tomorrow.

[1331] Server: Generates reminders and sends them to devices.

[1332] Device: If stress is low, notifications will be sent as normal, but if stress is high, notifications will be sent in a softer tone.

[1333] Prompt Sentence Examples

[1334] Prompt Sentence Examples

[1335] "Please explain in natural language the specific processing steps of the Desktop Magician system. Include the following items: obtaining user schedule data, automatically displaying materials, emotion recognition using an emotion engine, activity monitoring and alerts, and reminder notifications. Please clearly indicate the names and functions of the specific hardware and software used in all processing steps."

[1336] Specific questions

[1337] "After a user logs in, how does the server get the user's schedule data? And how is that data processed?"

[1338] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1339] Program processing flow and specific explanation

[1340] Step 1: User Login

[1341] User: Start the PC, click on the Desktop Magician app to go to the login screen, enter your username and password, and click the "Login" button.

[1342] Terminal: Sends the entered authentication information to the server.

[1343] Server: Compares the user information stored in the database, and if authentication is successful, generates an authentication token and sends it to the terminal.

[1344] Input: Username, Password

[1345] Output: Authentication token

[1346] Step 2: Retrieving and saving schedule data

[1347] Server: If authentication is successful, the external calendar API is used to retrieve the user's schedule data, including the event date, time, title, and details.

[1348] Server: Analyzes the acquired schedule data and stores it in a database.

[1349] Input: Schedule data from an external calendar API

[1350] Output: Schedule data stored in the server database

[1351] Step 3: Synchronize schedule data

[1352] Terminal: The Desktop Magician application running on the user's PC sends a schedule data update request to the server at regular intervals (e.g., every 5 minutes).

[1353] Server: Sends the latest schedule data to the terminal.

[1354] Terminal: Save the received schedule data in local storage.

[1355] Input: Schedule data update request from the server

[1356] Output: The latest schedule data saved in local storage

[1357] Step 4: Automatically display required materials

[1358] Device: Check your next appointment based on the current time. For example, if you have a meeting scheduled for 10:00 AM, identify the materials related to that meeting at 9:45 AM.

[1359] Device: The identified document will be displayed as a pop-up in a prominent location on your desktop.

[1360] Input: Schedule data in local storage

[1361] Output: Related documents displayed on your desktop

[1362] Step 5: Emotion Recognition

[1363] Device: Captures the user's facial expressions and voice in real time through the built-in camera and microphone.

[1364] Emotion engine: Analyzes acquired data using a deep learning model to evaluate the user's emotional state.

[1365] Device: Dynamically adjusts desktop display content and notification timing based on evaluation results.

[1366] Input: Data from the built-in camera and microphone

[1367] Output: Evaluated user emotional state and adjusted desktop display content

[1368] Step 6: Activity monitoring and alerts

[1369] Terminal: Monitors the title bar of the active window to see if the user is working according to a schedule.

[1370] Terminal: If the user is working outside of their schedule, a warning popup will appear after a specified time (e.g. 5 minutes).

[1371] Emotion engine: If the user's emotional state is high stress, the warning content will be softened.

[1372] Input: Active window title bar information

[1373] Output: Warning popup content

[1374] Step 7: Reminders

[1375] Server: Analyzes schedule data and generates reminders when important appointments or deadlines are approaching.

[1376] Server: Sends the generated reminders to the device.

[1377] Device: Notifies the user of received reminders and adjusts the content and timing using the emotion engine. For example, it sends a subtle notification such as "Remember the project submission deadline tomorrow."

[1378] Input: Schedule data

[1379] Output: Reminders notified to the user

[1380] (Application example 2)

[1381] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1382] In logistics center operations, managing schedules and related documents is essential for employees to work efficiently. However, doing this manually takes time and is prone to errors, which can lead to reduced work efficiency. Furthermore, long working hours and a stressful work environment can also put employees at risk of suffering from mental health issues, so a solution to these issues is needed.

[1383] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring user schedule data, means for identifying necessary materials based on the acquired schedule data, means for automatically displaying the identified materials on a display device on the user's terminal, means for recognizing the user's emotion, and means for adjusting the display content based on the recognized emotion. This allows employees to proceed with their work based on the automatically displayed schedule and related materials, and the use of emotion recognition technology enables them to work while maintaining their mental health.

[1384] "User schedule data" is information about the user's work schedule and task times.

[1385] "Means of acquisition" refers to the methods and systems used to collect schedule data from external databases and APIs.

[1386] "Necessary materials" refers to documents and information related to schedule data, such as work instructions and picking lists, which are necessary for carrying out work.

[1387] "Means for identifying" refers to a method or system that analyzes schedule data to determine what materials are needed at that time.

[1388] "User's Device" refers to a computer, smartphone, smart glasses, or other electronic device used by a User to conduct business.

[1389] A "display device" is a screen or display that allows a user to visually view information.

[1390] "Means for automatic display" is a mechanism for automatically displaying necessary materials on the display device of a terminal under certain conditions.

[1391] "Means for recognizing emotions" refers to technologies or systems that analyze the user's facial expressions and vocal characteristics to infer their emotional state at that time.

[1392] The "means for adjusting the displayed content" refers to a method or system for changing the type of information displayed or the way it is expressed based on the recognized emotion.

[1393] The "activity monitoring means" is a system for observing and recording the user's work status in real time.

[1394] The "means for displaying warnings" is a mechanism for displaying warning messages on the terminal display when the user's current activity deviates from the schedule or when a problem occurs.

[1395] The "means for generating reminders" is a notification function that notifies the user of important appointments and task deadlines based on schedule data.

[1396] The "means for displaying a reminder" is a system for displaying the generated reminder on the display device of the user's terminal.

[1397] "Means for adjusting reminder content and timing" refers to a system or method for changing the content of reminder messages and / or notification timing in consideration of a perceived emotional state.

[1398] This invention provides a "Logistics Magician" system for improving the work efficiency and mental health of employees at logistics centers. This system acquires the user's schedule data and automatically presents relevant materials based on that data, thereby improving work efficiency. It also uses emotion recognition technology to support the mental health of employees.

[1399] Basic system configuration

[1400] The basic configuration of the system is as follows:

[1401] 1. Server:

[1402] Obtaining schedule data: The server uses the API to obtain the user's schedule data from an external database.

[1403] Data management and storage: The acquired schedule data is stored in a database and the necessary data is sent to the terminal.

[1404] Generate reminders: Analyze schedule data and generate reminders.

[1405] 2. Terminal:

[1406] Schedule data synchronization: The terminal receives schedule data provided by the server and stores it in local storage.

[1407] Automatic display of materials: Based on the current time, schedule data and related materials are identified and displayed on the display device.

[1408] Activity monitoring and alerts: Monitors users' work activity and displays alerts if it deviates from the schedule.

[1409] Emotion engine: Using emotion recognition technology, it analyzes the user's emotional state and adjusts the display content and reminder content.

[1410] Specific processing explanation

[1411] 1. Data acquisition and display:

[1412] Hardware: Smart glasses, smartphones

[1413] The server retrieves the user's schedule data from an external database or API and stores it in the database. The device receives the latest schedule data from the server at regular intervals and stores it in local storage. When executed, the schedule data and related materials are displayed on the display device based on the current time.

[1414] Example: If a user has a meeting scheduled at 10:00 AM, materials related to that meeting will automatically appear on the display at 9:45 AM.

[1415] 2. Emotion recognition:

[1416] Software: OpenCV

[1417] The system analyzes the device's camera footage in real time and estimates the user's emotional state from their facial expressions, thereby assessing their stress level and fatigue.

[1418] Example: If the user feels fatigued, the device will notify them to "take a 5-minute break to relax."

[1419] 3. Notifications and Reminders:

[1420] It notifies users of important tasks and break times based on schedule data, and adjusts the content and timing of reminder messages based on emotional state.

[1421] Example: When a user is in a high-stress state, a reminder is displayed discreetly: "It's time for your next task. Please be mindful."

[1422] Prompt Sentence Examples

[1423] "If the user is not performing the current task accurately, display 'Are you continuing to work as planned?' and generate a prompt sentence recommending a break based on the emotion recognition results."

[1424] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1425] Step 1:

[1426] Acquiring and managing schedule data

[1427] Server: When a user logs in, the server uses an API to retrieve the user's schedule data from an external database, including the date, time, title, and details of the event.

[1428] Input: User ID, API authentication information.

[1429] Data processing: Call the API and receive schedule data in JSON format as a response.

[1430] Output: Schedule data in JSON format.

[1431] What happens: The server runs this process periodically and saves the schedule data to a database to keep it up to date.

[1432] Step 2:

[1433] Schedule data synchronization

[1434] Terminal: The terminal receives schedule data provided by the server at regular intervals and stores it in local storage.

[1435] Input: The request to the server.

[1436] Data calculation: Analyzes the JSON data sent from the server and saves it in local storage.

[1437] Output: The latest schedule data is saved locally.

[1438] Specific operation: The device sends a request to the server every five minutes to obtain the latest schedule data and update it to local storage.

[1439] Step 3:

[1440] Automatic display of materials

[1441] Terminal: Based on the current time, schedule data and related materials are identified and displayed on the display device.

[1442] Input: Local storage schedule data, current time.

[1443] Data calculation: Compare the current time with schedule data to identify the corresponding materials.

[1444] Output: Relevant material displayed on a display device.

[1445] What it does: If you have a meeting scheduled for 10:00 AM, presentations and documents related to that meeting will automatically appear at 9:45 AM.

[1446] Step 4:

[1447] emotion recognition

[1448] Device: Analyzes camera images from smart glasses or smartphones in real time and estimates the user's emotional state from their facial expressions.

[1449] Input: Camera footage.

[1450] Data calculation: Using OpenCV, emotions are estimated using a facial expression analysis model.

[1451] Output: Emotional state, such as stress level and fatigue level.

[1452] What it does: The device's camera captures the user's face and analyzes the image in real time to determine whether the user is tired.

[1453] Step 5:

[1454] Generate and coordinate notifications and reminders

[1455] Server (generation) and device (adjustment): The server generates reminders based on schedule data and sends them to the device. The device adjusts the content and timing of the reminders based on the recognized emotions.

[1456] Input: Schedule data, emotional state.

[1457] Data calculation: Analyze schedule data and adjust reminders based on emotional state.

[1458] Output: The adjusted reminder is displayed on the display device.

[1459] What it does: If the user is in a high-stress state, the reminder is subtly displayed as "It's time for your next task. Please be mindful."

[1460] Step 6:

[1461] Activity monitoring and alerts

[1462] Terminal: Monitors the user's work activity and displays a warning if it differs from the scheduled data.

[1463] Input: Current activity, schedule data.

[1464] Data calculation: Compares activity content with schedule data and generates a warning if there is a discrepancy.

[1465] Output: A warning message is displayed.

[1466] What it does: If a user is browsing a website other than the scheduled task, a message will pop up asking, "Are you still working as planned?"

[1467] Prompt Sentence Examples

[1468] "If the user is not performing the current task accurately, display 'Are you continuing to work as planned?' and generate a prompt sentence recommending a break based on the emotion recognition results."

[1469] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1470] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1471] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1472] [Fourth embodiment]

[1473] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1474] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1475] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1476] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1477] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1478] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1479] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1480] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1481] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1482] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1483] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1484] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1485] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1486] This invention is a desktop magician system for improving the user's work efficiency. It aims to acquire the user's schedule data and automatically display the necessary documents on the desktop based on that data. The system is realized by linking a server and a terminal.

[1487] System configuration

[1488] The basic system configuration is as follows:

[1489] 1. Server:

[1490] Retrieve user schedule data from external services (e.g., Google Calendar or Outlook Calendar).

[1491] The acquired schedule data is managed and saved, and the necessary data is sent to the user terminal.

[1492] Analyze schedule data and generate reminders.

[1493] 2. Terminal:

[1494] Schedule data provided by the server is received and stored in local storage.

[1495] Display the necessary materials on your desktop based on the current time.

[1496] Monitor users' work activity and display alerts when work is outside of schedule.

[1497] Notify users of reminders.

[1498] Overview of program processing

[1499] 1. Acquiring and saving schedule data

[1500] Server: When a user logs in, the server uses an API to retrieve the user's schedule data and saves it in a database. This data includes the date, time, title, and details of the event.

[1501] 2. Schedule data synchronization

[1502] Server: Updates the schedule data at regular intervals to keep it up to date.

[1503] Terminal: Receives schedule data updates from the server at regular intervals and saves them in local storage.

[1504] 3. Automatic display of materials

[1505] Terminal: Based on the current time, schedule data and related materials are identified and displayed on the desktop. For example, if a meeting is scheduled for 10:00 AM, materials related to that meeting will be displayed on the desktop 15 minutes before the meeting.

[1506] 4. Activity Monitoring and Alerts

[1507] Terminal: Monitors user activity and displays a warning pop-up after a certain period of time if the user is performing tasks that differ from the scheduled data. This pop-up prompts the user to recheck their schedule.

[1508] 5. Reminders

[1509] Server: Analyzes schedule data and generates reminders when important events or deadlines are approaching. Reminders are generated at set times (e.g., 3 days before, 1 day before).

[1510] On your device: Reminder notifications are received and displayed on your desktop. After you confirm the reminder, you can use the snooze feature to receive reminders again.

[1511] Specific examples

[1512] User's morning activities:

[1513] 1. A user starts up their PC in the morning and logs into Desktop Magician.

[1514] 2. Server: Obtains the user's latest schedule data from an external calendar service and sends it to the device.

[1515] 3. Terminal: Based on the received data, materials related to the meeting at 10:00 AM today are displayed on the desktop.

[1516] Monitoring during work:

[1517] 1. In the morning, the user is browsing the web for an activity other than the scheduled "report writing."

[1518] 2. Terminal: Monitor the title bar of the active window and display a warning popup if it is different from the schedule.

[1519] 3. A pop-up will appear saying, "You are currently working on something different from your schedule. Would you like to return to your schedule?" and the user will select "Yes."

[1520] Reminder:

[1521] 1. In the evening, a reminder appears on your device to let you know that a project is due tomorrow.

[1522] 2. Server: Generates reminders the day before and sends them to the device.

[1523] 3. Device: Reminders pop up on the desktop so users can take action.

[1524] As described above, the present invention provides an environment in which a user can work efficiently through a specific program and its processing.

[1525] The processing flow will be explained below.

[1526] Step 1:

[1527] The user logs in to Desktop Magician. Once logged in, the system sends the user authentication information to the server.

[1528] Step 2:

[1529] The server receives the user's authentication information and verifies that the user is a valid user. If authentication is successful, the server retrieves the user's schedule data (e.g., Google Calendar or Outlook Calendar) from an external API.

[1530] Step 3:

[1531] The server analyzes the acquired schedule data, extracts the user's schedule items, and stores them in a database, including the schedule date and time, title, details, and links to attachments.

[1532] Step 4:

[1533] The device requests synchronization of schedule data from the server at regular intervals (for example, every morning), and the server sends the latest schedule data to the device.

[1534] Step 5:

[1535] The device stores the received schedule data in local storage, then compares the current time with the schedule data to identify events scheduled within the next few hours.

[1536] Step 6:

[1537] The device will search for materials related to the identified event and automatically display them on the desktop. For example, if a meeting is scheduled for 10:00 a.m., materials related to this meeting will be displayed on the desktop 15 minutes before the meeting.

[1538] Step 7:

[1539] The device monitors the user's work activity and displays a warning popup after a specified time (e.g., 5 minutes) if the user is behaving differently from the scheduled work.

[1540] Step 8:

[1541] The device will display a pop-up warning to the user, saying, "You are currently working on something different from your scheduled schedule. Would you like to return to your scheduled schedule?" If the user selects "Yes," they will be provided with a link to easily return to their scheduled materials or applications.

[1542] Step 9:

[1543] The server periodically analyzes schedule data and generates reminders when important events or deadlines are approaching, which are then sent to the device.

[1544] Step 10:

[1545] The device will display the received reminder as a pop-up on the desktop, and after the user confirms the reminder, they can set the snooze function to be notified again later.

[1546] This series of processes allows users to concentrate on their work efficiently and makes schedule management easier.

[1547] Example 1

[1548] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1549] Conventional schedule management systems require users to prepare materials themselves, and do not provide sufficient warnings or reminders when they are working outside of the schedule, which can lead to reduced work efficiency.

[1550] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1551] In this invention, the server includes means for acquiring user schedule data from an external service, means for saving the acquired schedule data in a database, and means for periodically updating the schedule data to keep it up to date. This allows necessary materials to be automatically displayed on the desktop based on the latest schedule data, improving the user's work efficiency.

[1552] The terminal also includes means for identifying materials relevant to the current time based on the schedule data and automatically displaying them on the desktop, means for monitoring the user's activity during work and displaying a warning if the user is performing work that differs from the schedule, means for analyzing the schedule data and generating a reminder when an important appointment or deadline approaches, and means for displaying the generated reminder on the desktop. This enables the user to accurately perform the scheduled work and proceed with work without missing important appointments or deadlines.

[1553] "User schedule data" refers to information about plans and events that a user has registered in an external service (for example, a calendar service), and includes information such as date and time, title, and details.

[1554] "External service" refers to an online platform for managing, storing, and providing users' schedule data, such as a calendar application or a cloud-based schedule management service.

[1555] A "database" is a system that organizes, stores, and manages acquired schedule data.

[1556] "Regularly updated" means that schedule data is reacquired from an external service at regular intervals to keep the contents of the database up to date.

[1557] "Necessary materials" refers to documents and files related to scheduled tasks and events based on the user's schedule data.

[1558] "Automatically displaying on the desktop" means displaying materials on the screen of the user's terminal without the need for manual operation.

[1559] "Terminal" refers to an electronic computer that allows a user to use the Desktop Magician System, and includes desktop PCs, notebook PCs, tablets, etc.

[1560] "Activity" refers to the work or operation that a user performs on a terminal, and indicates the type of work that can be determined, for example, by the title bar of an active window.

[1561] A "warning popup" is a small window that displays a warning or instructions when the user is performing work that is different from the schedule.

[1562] A "reminder" is a message that notifies the user when an important event or deadline is approaching, and is generated at a specified timing.

[1563] "Local storage" refers to a storage device (such as a hard disk or SSD) built into the user's terminal, and is used to store schedule data and related materials received from the server.

[1564] This invention is a desktop magician system for improving the user's work efficiency. The system acquires the user's schedule data and automatically displays the necessary documents on the desktop based on that data. The system is realized by linking a server and a terminal.

[1565] server

[1566] When a user logs in to the Desktop Magician system, the server uses an API to retrieve the user's schedule data from external services (such as Google Calendar or Outlook Calendar). This retrieved data includes the date, time, title, and details of the appointment. The retrieved data is then stored in a database on the server. The server then updates the schedule data at regular intervals to keep it up to date. In addition, the server analyzes the schedule data and generates reminders when important appointments or deadlines are approaching, which are then sent to the device.

[1567] Terminal

[1568] The device periodically receives schedule data provided by the server and saves it in local storage. Based on the current time, it identifies the schedule data and related materials and automatically displays them on the desktop. The device also monitors the user's work activity and displays a warning pop-up after a certain period of time if the user is performing work that differs from the schedule data. This pop-up prompts the user to reconfirm their schedule. It also sends reminder notifications, providing information for the user to take action. Reminder notifications are displayed as a pop-up on the desktop, and after the user confirms the reminder, they can use the snooze function to be notified again.

[1569] Usage example

[1570] User's morning activities:

[1571] A user starts up their PC in the morning and logs into Desktop Magician.

[1572] The server obtains the user's latest schedule data from an external calendar service and sends it to the terminal.

[1573] Based on the data received by the device, materials related to the meeting at 10:00 a.m. today will be displayed on the desktop.

[1574] Monitoring during work:

[1575] In the morning, the user is browsing the web, which is different from the scheduled "report writing" activity.

[1576] The terminal monitors the title bar of the active window and displays a warning popup if it is different from what is expected.

[1577] The popup displays "You are currently working on something different from your schedule. Would you like to return to your schedule?" and the user selects "Yes."

[1578] Reminder:

[1579] In the evening, a reminder appears on your device to let you know that a project is due tomorrow.

[1580] The server generates a reminder the day before and sends it to the device.

[1581] The device will pop up a reminder on your desktop so you can take action.

[1582] Example of input prompt for generative AI model

[1583] "A user logs into the Desktop Magician system. I want the documents related to the 10 AM meeting to be displayed on the desktop. I also want the system to monitor the user's work activity and warn them if they are working in a different way."

[1584] As described above, the present invention improves the efficiency of a user's work by effectively managing the user's schedule through cooperation between the server and the terminal and providing necessary materials at the appropriate time.

[1585] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1586] Step 1: User Login

[1587] Input: User ID and password

[1588] Process: The user starts up the PC and logs in to the Desktop Magician system. The entered user ID and password are sent to the server for authentication.

[1589] Data processing: The server authenticates the user ID and password and generates a success or failure result.

[1590] Output: If authentication is successful, obtain the user's basic information and authorization information and send it to the terminal. If authentication is unsuccessful, display an error message.

[1591] Specific operation: The user enters the user ID and password on the login screen and clicks the login button. The server performs authentication, and if successful, the main screen is displayed.

[1592] Step 2: Retrieving and saving schedule data

[1593] Input: User login information

[1594] Processing: Based on the user's login information, the server uses an API to retrieve schedule data from an external calendar service (e.g., Google Calendar, Outlook Calendar).

[1595] Data processing: The acquired schedule data includes the appointment date, time, title, and details, and is saved in a database on the server.

[1596] Output: Schedule data stored in the database

[1597] What happens: The server sends an API request using the user's calendar account information to retrieve and store schedule data.

[1598] Step 3: Synchronize schedule data

[1599] Input: API request to external calendar service

[1600] Processing: The server periodically sends an API request to the external calendar service to retrieve the latest schedule data.

[1601] Data processing: Compare new or changed schedule data with the database and update it to the latest version.

[1602] Output: Updated database schedule data

[1603] Specific operation: The server makes an API request every hour or at a set interval and updates the database with the retrieved data.

[1604] Step 4: Send schedule data

[1605] Input: Schedule data from the database

[1606] Processing: The terminal periodically sends a request to the server and receives the latest schedule data.

[1607] Data processing: Save the received schedule data to local storage.

[1608] Output: Schedule data in local storage

[1609] Specific operation: The device requests schedule data from the server every 30 minutes and saves the received data.

[1610] Step 5: Automatically display materials

[1611] Input: Local storage schedule data and current time

[1612] Processing: Based on the schedule data, the terminal identifies materials relevant to the current time and displays them on the desktop.

[1613] Data processing: Analyze schedule data and obtain paths to identified materials.

[1614] Output: Documents displayed on the desktop

[1615] Specific operation: The device checks the current time, and if there is a meeting at 10:00, the meeting materials will be displayed on the desktop at 9:45.

[1616] Step 6: Activity monitoring and alerts

[1617] Input: User activity data (active window title bar)

[1618] Action: The device monitors the user's activity and displays a warning pop-up if the user is performing an activity that differs from the scheduled data.

[1619] Data processing: Analyze activity data and compare it with planned work.

[1620] Output: Warning popup

[1621] Specific behavior: If the user is browsing the web in a way that is not part of their schedule, the message "You are currently working on something different from your schedule. Would you like to return to your schedule?" will be displayed.

[1622] Step 7: Reminders

[1623] Input: Schedule data from the database

[1624] Processing: The server analyzes the schedule data and generates reminders when important events or deadlines approach.

[1625] Data processing: Identify important events and deadlines and generate reminder messages.

[1626] Output: The generated reminder is sent to the device and displayed on the desktop.

[1627] What happens: The server generates a reminder for the previous day, and the device displays the reminder on the desktop.

[1628] Example of input prompt for generative AI model

[1629] "A user logs into the Desktop Magician system. I want the documents related to the 10 AM meeting to be displayed on the desktop. I also want the system to monitor the user's work activity and warn them if they are working in a different way."

[1630] (Application example 1)

[1631] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1632] Conventional factory robots did not properly manage work schedules or monitor work status in real time, making it difficult to achieve efficient production activities. Also, when workers performed tasks that deviated from the schedule, appropriate measures could not be taken. Furthermore, the lack of reminders for important tasks and deadlines meant that efficient work operations could not be achieved.

[1633] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1634] In this invention, the server includes a means for acquiring a user's schedule data, a means for identifying necessary documents based on the acquired schedule data, and a means for automatically displaying the identified documents on the desktop of the user's terminal. This enables effective schedule management and real-time monitoring of work status, which was difficult with conventional factory robots. It also enables warnings when work is being performed that deviates from the schedule and reminders for important work and deadlines, thereby achieving efficient work management.

[1635] "User schedule data" refers to information relating to the date, time, schedule contents, etc. that the user has planned in advance.

[1636] "Means of acquisition" refers to application programs or APIs for acquiring user schedule data from external calendar services, etc.

[1637] "Required materials" refers to documents and digital files that are required to perform a specific task based on the schedule data.

[1638] "Means for identification" refers to algorithms or software that analyze acquired schedule data and identify necessary materials based on that data.

[1639] "Means for automatic display" refers to a program or tool for automatically displaying the identified material on the user's terminal.

[1640] A "factory robot" refers to an automated mechanical device designed to perform production tasks in a factory.

[1641] "Current activity" refers to a specific task or activity that a user or factory robot is currently performing.

[1642] "Monitoring means" refers to programs and sensors that observe and record the work activities of users or factory robots in real time.

[1643] The "means for displaying a warning" refers to a program or device for displaying a warning message when a user or a factory robot is performing an activity that differs from the schedule.

[1644] A "reminder" is a message that notifies a user when an important appointment or deadline is approaching, based on the user's schedule data.

[1645] "Means for generating" refers to programs or software for generating reminders based on schedule data.

[1646] "Means for monitoring progress" refers to programs and sensors that monitor the progress of tasks being performed by factory robots in real time.

[1647] "Efficient work instructions" refers to specific instructions and guidance provided to factory robots to help them work more efficiently.

[1648] This invention is a system for improving the work efficiency of factory robots. This system acquires user schedule data, prepares necessary materials and tools based on that data, and provides them to the factory robots at the appropriate time, thereby improving work efficiency.

[1649] System configuration

[1650] The basic system configuration is as follows:

[1651] 1. Server:

[1652] Obtain user schedule data from an external service (e.g., a calendar service).

[1653] The acquired schedule data is managed and saved, and the necessary data is sent to the user's device or factory robot.

[1654] Analyze schedule data and generate reminders.

[1655] 2. Terminal:

[1656] Schedule data provided by the server is received and stored in local storage.

[1657] Display required materials based on the current time.

[1658] 3. Factory robots:

[1659] Automatically prepare materials and tools.

[1660] Monitor work activity and display alerts when schedules deviate.

[1661] Receive reminders and provide work instructions.

[1662] Monitor progress and provide efficient work instructions.

[1663] Overview of program processing

[1664] 1. Acquiring and saving schedule data

[1665] Server: When a user logs in, the server uses an API to retrieve the user's schedule data and saves it in a database. This data includes the date, time, and content of the schedule.

[1666] 2. Schedule data synchronization

[1667] Server: Updates the schedule data at regular intervals to keep it up to date.

[1668] Terminal: Receives schedule data updates from the server at regular intervals and saves them in local storage.

[1669] 3. Automatic display of materials

[1670] Terminal: Based on the current time, schedule data and related materials are identified and displayed on the desktop. For example, if a welding job is scheduled for 10:00 AM, materials related to that job will be displayed on the desktop 15 minutes beforehand.

[1671] 4. Activity Monitoring and Alerts

[1672] Factory robots: Monitor activities and display a warning pop-up after a certain time if the robot is performing work that is different from the schedule, prompting the worker to recheck the schedule.

[1673] 5. Reminders

[1674] Server: Analyzes schedule data and generates reminders when important events or deadlines are approaching. Reminders are generated at a set time (e.g., one day before).

[1675] Factory robots: receive reminder notifications and provide work instructions.

[1676] Specific examples

[1677] Consider a scenario in which a specific factory robot is scheduled to perform welding work starting at 10:00 AM. When a user logs into the system in the morning, the server retrieves the user's schedule data from an external calendar service and sends it to the terminal and factory robot. The terminal displays materials related to the 10:00 AM welding work on its desktop, and the factory robot prepares the tools necessary for welding. If the robot is performing a different activity during the work, a warning is displayed and the user is asked to confirm. Furthermore, the server notifies the user that the work deadline is one day away, generates a reminder, and provides instructions to the factory robot.

[1678] Prompt Sentence Examples

[1679] To check the user's morning schedule, retrieve the factory robot's schedule data and display the materials related to the 10:00 a.m. work. Also, if there is work outside the schedule, display a warning and generate a reminder within 24 hours.

[1680] In this way, the present invention improves the work efficiency of factory robots through specific programs and means, and realizes effective schedule management and real-time work monitoring.

[1681] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1682] Step 1:

[1683] When a user logs in, the server accesses the API of the external calendar service to obtain the user's schedule data. Specifically, it uses the user ID as an API parameter and sends a request. Based on this input, the schedule data (schedule date and time, title, and details) is obtained in JSON format. The obtained data is saved in the server's database.

[1684] Step 2:

[1685] The server updates the schedule data at regular intervals. To update, it accesses the API of the external calendar service again to obtain the latest schedule data. If new data exists, it updates the database contents. This input data keeps the server database up to date with the latest schedule data.

[1686] Step 3:

[1687] The terminal periodically obtains schedule data updates from the server. The terminal receives the latest schedule data from the server and stores the data in the terminal's local storage. This input data allows the terminal to access the latest schedule data.

[1688] Step 4:

[1689] The terminal references the schedule data based on the current time and identifies the relevant materials 15 minutes before the next scheduled appointment. For example, if welding work is scheduled for 10:00 a.m., the materials needed for that work will be identified and displayed on the desktop. Based on this input, the file path of the identified materials is output, and the materials are displayed on the desktop.

[1690] Step 5:

[1691] The factory robot automatically prepares the identified materials and tools. Based on the schedule data, it generates a list of tools required for the day's work and prepares them. For example, in the case of welding work, welding machines and protective equipment are prepared. This input data allows for efficient work preparation.

[1692] Step 6:

[1693] Factory robots use sensors and programs to monitor their work activities. They observe the current work status in real time and compare it with the schedule data. If the work differs from the schedule, a warning pop-up is displayed. This input data detects discrepancies and outputs a warning to the user or robot.

[1694] Step 7:

[1695] The server analyzes the schedule data and generates reminders when important appointments or deadlines are approaching. These reminders are generated at a set time (e.g., one day before) based on the schedule date and time. The generated reminders are sent to the user's device and factory robots. This input data allows the reminders to be output at the appropriate time.

[1696] Step 8:

[1697] The factory robot receives the reminder notification and provides the user with work instructions. For example, a notification is received the day before welding work and an instruction is issued to start preparations the next day. Based on this input data, the reminder content is output and the factory robot is notified reliably.

[1698] This enables effective schedule management, real-time monitoring of work status, and efficient provision of work instructions, which were difficult with conventional factory robots.

[1699] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1700] This invention is a desktop magician system for improving the user's work efficiency and mental well-being. It acquires the user's schedule data and automatically displays necessary documents on the desktop based on that data. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized support.

[1701] System configuration

[1702] The basic system configuration is as follows:

[1703] 1. Server:

[1704] Obtaining user schedule data from an external service (e.g., a calendar API).

[1705] The acquired schedule data is managed and saved, and the necessary data is sent to the user terminal.

[1706] Analyze schedule data and generate reminders.

[1707] 2. Terminal:

[1708] Schedule data provided by the server is received and stored in local storage.

[1709] Display the necessary materials on your desktop based on the current time.

[1710] Monitor users' work activity and display alerts when work is outside of schedule.

[1711] Notify users of reminders.

[1712] It has an emotion engine that recognizes the user's emotions and adjusts desktop displays, alerts, and reminder notifications based on the user's emotional state.

[1713] Overview of program processing

[1714] 1. Acquiring and saving schedule data

[1715] Server: When a user logs in, the server uses an API to retrieve the user's schedule data and saves it in a database. This data includes the date, time, title, and details of the event.

[1716] 2. Schedule data synchronization

[1717] Server: Updates the schedule data at regular intervals to keep it up to date.

[1718] Terminal: Receives schedule data updates from the server at regular intervals and saves them in local storage.

[1719] 3. Automatic display of materials

[1720] Terminal: Based on the current time, schedule data and related materials are identified and displayed on the desktop. For example, if a meeting is scheduled for 10:00 AM, materials related to that meeting will be displayed on the desktop 15 minutes before the meeting.

[1721] 4. Emotion Recognition by Emotion Engine

[1722] Device: While the user is using the PC, the emotion engine processes input data from the built-in camera and microphone and analyzes the user's emotional state in real time. The emotional state is expressed as parameters such as stress level, excitement level, and concentration.

[1723] 5. Activity Monitoring and Alerts

[1724] Terminal: Monitors user activity and displays a warning pop-up after a specified time (e.g., 5 minutes) if the user is performing tasks that differ from the scheduled data. If the user is feeling overly stressed, the emotion engine can soften or postpone the warning.

[1725] 6. Reminders

[1726] Server: Analyzes schedule data and generates reminders when important events or deadlines are approaching. The generated reminders are sent to the device.

[1727] On-device: The emotional engine takes into account the user's emotional state and adjusts the content and timing of reminders. For example, if the user is feeling stressed, the reminder will be more toned down.

[1728] Specific examples

[1729] User's morning activities:

[1730] 1. A user starts up their PC in the morning and logs into Desktop Magician.

[1731] 2. Server: Obtains the user's latest schedule data from an external calendar service and sends it to the device.

[1732] 3. Terminal: Based on the received data, materials related to the meeting at 10:00 AM today are displayed on the desktop.

[1733] Monitoring during work:

[1734] 1. In the morning, the user is browsing the web for an activity other than the scheduled "report writing."

[1735] 2. Terminal: Monitor the title bar of the active window and display a warning popup if it is different from the schedule.

[1736] 3. The popup will say "You are currently working on something different from your schedule. Would you like to return to your schedule?" If the user is feeling annoyed, the warning will be more subtle.

[1737] Emotion engine response:

[1738] 1. When a user continues working for a long time, input from the camera and microphone indicates that stress is high.

[1739] 2. Device: Based on the emotion engine, notify "Take a 5-minute break to relax."

[1740] Reminder:

[1741] 1. In the evening, a reminder appears on your device to let you know that a project is due tomorrow.

[1742] 2. Server: Generates reminders the day before and sends them to the device.

[1743] 3. Device: If the user's stress level is low, the emotion engine will notify them normally, but if it is high, it will notify them in a softer way.

[1744] As described above, the present invention enables users to perform their work efficiently and in a mentally healthy environment through a specific program and its processing that takes into account the user's emotional state.

[1745] The processing flow will be explained below.

[1746] Step 1:

[1747] A user logs in to Desktop Magician. Upon login, the system sends the user's authentication information to the server.

[1748] Step 2:

[1749] The server receives the user's authentication information and verifies that the user is a valid user. If authentication is successful, the server retrieves the user's schedule data from an external API (e.g., a calendar API) and saves it in a database.

[1750] Step 3:

[1751] The server analyzes the schedule data and extracts the user's schedule items (such as the schedule date and time, title, details, and attachment links).

[1752] Step 4:

[1753] The device requests synchronization of schedule data from the server at regular intervals (for example, every morning), and the server sends the latest schedule data to the device.

[1754] Step 5:

[1755] The device saves the received schedule data in local storage. After saving, it compares the current time with the schedule data to identify upcoming events.

[1756] Step 6:

[1757] The device will search for documents related to the specified appointment and automatically display them on the desktop. For example, if a meeting is scheduled for 10:00 a.m., the relevant documents will be displayed on the desktop 15 minutes before the meeting.

[1758] Step 7:

[1759] The device collects data from the camera and microphone built into the user's PC and analyzes it with an emotion engine, which evaluates the user's emotional state (stress, excitement, concentration, etc.) in real time.

[1760] Step 8:

[1761] The device monitors the user's work activity and if the work being performed differs from the scheduled data, a warning pop-up will be displayed after a specified time (e.g., 5 minutes). The content of the warning is adjusted by the emotion engine.

[1762] Step 9:

[1763] The device displays a pop-up warning: "You are currently working on something other than your scheduled schedule. Would you like to return to your scheduled schedule?" If the user's emotional state is stressed, the message will be displayed in a more gentle manner.

[1764] Step 10:

[1765] The server periodically analyzes schedule data and generates reminders when important events or deadlines are approaching, and the generated reminders are sent to the device.

[1766] Step 11:

[1767] Reminders received by the device are displayed as a pop-up on the desktop. The emotion engine adjusts the content and timing of the reminder according to the user's emotional state. For example, if the user is in a high-stress state, the reminder will be displayed at a more discreet time.

[1768] This series of processes enables users to carry out their work efficiently and also to receive support with optimal timing and expression based on their emotional state.

[1769] Example 2

[1770] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1771] Conventional schedule management systems often provide a function to retrieve schedule data and display necessary documents on the user's desktop to improve work efficiency. However, because they do not take the user's emotional state into consideration, excessive work can cause mental strain and reduce work efficiency. In addition, warnings and reminder notifications when the user is working outside of the schedule are displayed in a uniform manner, which makes it difficult to respond flexibly to individual situations.

[1772] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1773] In this invention, the server includes means for acquiring user schedule data, means for identifying necessary materials based on the acquired schedule data, means for automatically displaying the identified materials on the desktop of the user's terminal, means for recognizing the user's emotional state, and means for adjusting the display content of the desktop based on the recognized emotional state. This makes it possible to improve work efficiency and reduce mental burden by taking the user's emotional state into consideration.

[1774] "User schedule data" refers to information including the date, time, title, details, etc. of schedules that a user manages through an external calendar service or manual input.

[1775] The "means of acquisition" refers to a function by which the server collects user schedule data using a means such as an external calendar API.

[1776] The "means for identifying necessary materials" refers to an algorithm or program for identifying related materials or information based on the acquired schedule data.

[1777] The "means for automatic display" is a function for automatically displaying the identified material on the desktop of the user's terminal at the required timing.

[1778] The "means for recognizing the user's emotional state" is a system that uses an emotion analysis algorithm to evaluate the user's emotional state based on data acquired from the built-in camera and microphone.

[1779] The "means for adjusting desktop display content" is a function that dynamically changes the content and timing of information and notifications displayed on the desktop based on the recognized emotional state of the user.

[1780] The "means for monitoring activity" is a system that monitors active windows running on a terminal and user operations, and determines whether or not they are in accordance with a schedule.

[1781] The "means for displaying a warning" is a function that displays a notification or a pop-up to call the user's attention when the user is performing work outside of the schedule.

[1782] The "means for generating reminders" is a function for creating reminders to notify the user of important appointments and deadlines based on schedule data.

[1783] The "means for adjusting the content and timing of reminders" is a system that changes the content of reminder messages and the timing of notifications based on the user's emotional state.

[1784] This invention is a desktop magician system that improves users' work efficiency and mental well-being. It is mainly composed of a server and a user terminal, each of which performs specific functions to achieve performance optimization through schedule management and emotion recognition. Specific implementation methods are described below.

[1785] Basic system configuration

[1786] 1. Server Functions

[1787] Retrieving schedule data: The server retrieves the user's schedule data using the external calendar API. The retrieved data includes the date, time, title, and details of the event.

[1788] Data storage and management: The acquired schedule data is stored in a database on the server and kept accessible upon user request.

[1789] Reminder generation: The server analyzes the schedule data and generates reminders when important appointments or deadlines are approaching, and sends them to the user's terminal.

[1790] 2. Device Features

[1791] Data synchronization and storage: The terminal periodically receives schedule data sent from the server and stores it in local storage.

[1792] Automatic display of necessary materials: Based on the current time, schedule data and related materials are identified and displayed on the desktop. For example, if a meeting is scheduled, materials related to that meeting are automatically displayed.

[1793] Emotion Recognition: The emotion engine analyzes the user's emotional state in real time through the device's built-in camera and microphone. The emotion engine evaluates stress levels, concentration, and other factors and adjusts the content displayed on the desktop accordingly.

[1794] Activity monitoring and warnings: The device monitors the user's work activity and displays a warning pop-up if it detects any deviation from the scheduled activity. It also has the ability to adjust the warning content according to the user's emotional state.

[1795] Reminder notification: Notifies the user of reminders and adjusts the content and timing of notifications using an emotion engine.

[1796] Hardware and software used

[1797] Calendar API: Use an external calendar service (e.g., Google Calendar API) to retrieve schedule data.

[1798] Database: A database system (e.g., MySQL) for storing and managing schedule data on the server.

[1799] Emotion engine: Software that runs deep learning models to analyze a user's facial expressions and voice.

[1800] Built-in camera and microphone: Installed on the user device to collect data for emotion recognition.

[1801] Specific examples

[1802] 1. User's morning activities

[1803] The user starts up the PC and logs into Desktop Magician.

[1804] Server: Obtains the latest schedule data from an external calendar service and sends it to the device.

[1805] Device: Display materials related to the 10:00 AM meeting on your desktop.

[1806] 2. Monitoring during work

[1807] The user is browsing the web for a purpose other than "writing reports."

[1808] On your device: Display a warning popup saying "You are currently working on something other than your scheduled schedule. Would you like to return to your scheduled schedule?"

[1809] 3. Emotional engine response

[1810] When working for long periods of time, input from the camera and microphone is used to determine high stress levels.

[1811] Device: Display a notification that says "Take a 5-minute break to relax."

[1812] 4. Reminders

[1813] In the evening, a reminder appears on your device to let you know that a project is due tomorrow.

[1814] Server: Generates reminders and sends them to devices.

[1815] Device: If stress is low, notifications will be sent as normal, but if stress is high, notifications will be sent in a softer tone.

[1816] Prompt Sentence Examples

[1817] Prompt Sentence Examples

[1818] "Please explain in natural language the specific processing steps of the Desktop Magician system. Include the following items: obtaining user schedule data, automatically displaying materials, emotion recognition using an emotion engine, activity monitoring and alerts, and reminder notifications. Please clearly indicate the names and functions of the specific hardware and software used in all processing steps."

[1819] Specific questions

[1820] "After a user logs in, how does the server get the user's schedule data? And how is that data processed?"

[1821] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1822] Program processing flow and specific explanation

[1823] Step 1: User Login

[1824] User: Start the PC, click on the Desktop Magician app to go to the login screen, enter your username and password, and click the "Login" button.

[1825] Terminal: Sends the entered authentication information to the server.

[1826] Server: Compares the user information stored in the database, and if authentication is successful, generates an authentication token and sends it to the terminal.

[1827] Input: Username, Password

[1828] Output: Authentication token

[1829] Step 2: Retrieving and saving schedule data

[1830] Server: If authentication is successful, the external calendar API is used to retrieve the user's schedule data, including the event date, time, title, and details.

[1831] Server: Analyzes the acquired schedule data and stores it in a database.

[1832] Input: Schedule data from an external calendar API

[1833] Output: Schedule data stored in the server database

[1834] Step 3: Synchronize schedule data

[1835] Terminal: The Desktop Magician application running on the user's PC sends a schedule data update request to the server at regular intervals (e.g., every 5 minutes).

[1836] Server: Sends the latest schedule data to the terminal.

[1837] Terminal: Save the received schedule data in local storage.

[1838] Input: Schedule data update request from the server

[1839] Output: The latest schedule data saved in local storage

[1840] Step 4: Automatically display required materials

[1841] Device: Check your next appointment based on the current time. For example, if you have a meeting scheduled for 10:00 AM, identify the materials related to that meeting at 9:45 AM.

[1842] Device: The identified document will be displayed as a pop-up in a prominent location on your desktop.

[1843] Input: Schedule data in local storage

[1844] Output: Related documents displayed on your desktop

[1845] Step 5: Emotion Recognition

[1846] Device: Captures the user's facial expressions and voice in real time through the built-in camera and microphone.

[1847] Emotion engine: Analyzes acquired data using a deep learning model to evaluate the user's emotional state.

[1848] Device: Dynamically adjusts desktop display content and notification timing based on evaluation results.

[1849] Input: Data from the built-in camera and microphone

[1850] Output: Evaluated user emotional state and adjusted desktop display content

[1851] Step 6: Activity monitoring and alerts

[1852] Terminal: Monitors the title bar of the active window to see if the user is working according to a schedule.

[1853] Terminal: If the user is working outside of their schedule, a warning popup will appear after a specified time (e.g. 5 minutes).

[1854] Emotion engine: If the user's emotional state is high stress, the warning content will be softened.

[1855] Input: Active window title bar information

[1856] Output: Warning popup content

[1857] Step 7: Reminders

[1858] Server: Analyzes schedule data and generates reminders when important appointments or deadlines are approaching.

[1859] Server: Sends the generated reminders to the device.

[1860] Device: Notifies the user of received reminders and adjusts the content and timing using the emotion engine. For example, it sends a subtle notification such as "Remember the project submission deadline tomorrow."

[1861] Input: Schedule data

[1862] Output: Reminders notified to the user

[1863] (Application example 2)

[1864] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1865] In logistics center operations, managing schedules and related documents is essential for employees to work efficiently. However, doing this manually takes time and is prone to errors, which can lead to reduced work efficiency. Furthermore, long working hours and a stressful work environment can also put employees at risk of suffering from mental health issues, so a solution to these issues is needed.

[1866] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring user schedule data, means for identifying necessary materials based on the acquired schedule data, means for automatically displaying the identified materials on a display device on the user's terminal, means for recognizing the user's emotion, and means for adjusting the display content based on the recognized emotion. This allows employees to proceed with their work based on the automatically displayed schedule and related materials, and the use of emotion recognition technology enables them to work while maintaining their mental health.

[1867] "User schedule data" is information about the user's work schedule and task times.

[1868] "Means of acquisition" refers to the methods and systems used to collect schedule data from external databases and APIs.

[1869] "Necessary materials" refers to documents and information related to schedule data, such as work instructions and picking lists, which are necessary for carrying out work.

[1870] "Means for identifying" refers to a method or system that analyzes schedule data to determine what materials are needed at that time.

[1871] "User's Device" refers to a computer, smartphone, smart glasses, or other electronic device used by a User to conduct business.

[1872] A "display device" is a screen or display that allows a user to visually view information.

[1873] "Means for automatic display" is a mechanism for automatically displaying necessary materials on the display device of a terminal under certain conditions.

[1874] "Means for recognizing emotions" refers to technologies or systems that analyze the user's facial expressions and vocal characteristics to infer their emotional state at that time.

[1875] The "means for adjusting the displayed content" refers to a method or system for changing the type of information displayed or the way it is expressed based on the recognized emotion.

[1876] The "activity monitoring means" is a system for observing and recording the user's work status in real time.

[1877] The "means for displaying warnings" is a mechanism for displaying warning messages on the terminal display when the user's current activity deviates from the schedule or when a problem occurs.

[1878] The "means for generating reminders" is a notification function that notifies the user of important appointments and task deadlines based on schedule data.

[1879] The "means for displaying a reminder" is a system for displaying the generated reminder on the display device of the user's terminal.

[1880] "Means for adjusting reminder content and timing" refers to a system or method for changing the content of reminder messages and / or notification timing in consideration of a perceived emotional state.

[1881] This invention provides a "Logistics Magician" system for improving the work efficiency and mental health of employees at logistics centers. This system acquires the user's schedule data and automatically presents relevant materials based on that data, thereby improving work efficiency. It also uses emotion recognition technology to support the mental health of employees.

[1882] Basic system configuration

[1883] The basic configuration of the system is as follows:

[1884] 1. Server:

[1885] Obtaining schedule data: The server uses the API to obtain the user's schedule data from an external database.

[1886] Data management and storage: The acquired schedule data is stored in a database and the necessary data is sent to the terminal.

[1887] Generate reminders: Analyze schedule data and generate reminders.

[1888] 2. Terminal:

[1889] Schedule data synchronization: The terminal receives schedule data provided by the server and stores it in local storage.

[1890] Automatic display of materials: Based on the current time, schedule data and related materials are identified and displayed on the display device.

[1891] Activity monitoring and alerts: Monitors users' work activity and displays alerts if it deviates from the schedule.

[1892] Emotion engine: Using emotion recognition technology, it analyzes the user's emotional state and adjusts the display content and reminder content.

[1893] Specific processing explanation

[1894] 1. Data acquisition and display:

[1895] Hardware: Smart glasses, smartphones

[1896] The server retrieves the user's schedule data from an external database or API and stores it in the database. The device receives the latest schedule data from the server at regular intervals and stores it in local storage. When executed, the schedule data and related materials are displayed on the display device based on the current time.

[1897] Example: If a user has a meeting scheduled at 10:00 AM, materials related to that meeting will automatically appear on the display at 9:45 AM.

[1898] 2. Emotion recognition:

[1899] Software: OpenCV

[1900] The system analyzes the device's camera footage in real time and estimates the user's emotional state from their facial expressions, thereby assessing their stress level and fatigue.

[1901] Example: If the user feels fatigued, the device will notify them to "take a 5-minute break to relax."

[1902] 3. Notifications and Reminders:

[1903] It notifies users of important tasks and break times based on schedule data, and adjusts the content and timing of reminder messages based on emotional state.

[1904] Example: When a user is in a high-stress state, a reminder is displayed discreetly: "It's time for your next task. Please be mindful."

[1905] Prompt Sentence Examples

[1906] "If the user is not performing the current task accurately, display 'Are you continuing to work as planned?' and generate a prompt sentence recommending a break based on the emotion recognition results."

[1907] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1908] Step 1:

[1909] Acquiring and managing schedule data

[1910] Server: When a user logs in, the server uses an API to retrieve the user's schedule data from an external database, including the date, time, title, and details of the event.

[1911] Input: User ID, API authentication information.

[1912] Data processing: Call the API and receive schedule data in JSON format as a response.

[1913] Output: Schedule data in JSON format.

[1914] What happens: The server runs this process periodically and saves the schedule data to a database to keep it up to date.

[1915] Step 2:

[1916] Schedule data synchronization

[1917] Terminal: The terminal receives schedule data provided by the server at regular intervals and stores it in local storage.

[1918] Input: The request to the server.

[1919] Data calculation: Analyzes the JSON data sent from the server and saves it in local storage.

[1920] Output: The latest schedule data is saved locally.

[1921] Specific operation: The device sends a request to the server every five minutes to obtain the latest schedule data and update it to local storage.

[1922] Step 3:

[1923] Automatic display of materials

[1924] Terminal: Based on the current time, schedule data and related materials are identified and displayed on the display device.

[1925] Input: Local storage schedule data, current time.

[1926] Data calculation: Compare the current time with schedule data to identify the corresponding materials.

[1927] Output: Relevant material displayed on a display device.

[1928] What it does: If you have a meeting scheduled for 10:00 AM, presentations and documents related to that meeting will automatically appear at 9:45 AM.

[1929] Step 4:

[1930] emotion recognition

[1931] Device: Analyzes camera images from smart glasses or smartphones in real time and estimates the user's emotional state from their facial expressions.

[1932] Input: Camera footage.

[1933] Data calculation: Using OpenCV, emotions are estimated using a facial expression analysis model.

[1934] Output: Emotional state, such as stress level and fatigue level.

[1935] What it does: The device's camera captures the user's face and analyzes the image in real time to determine whether the user is tired.

[1936] Step 5:

[1937] Generate and coordinate notifications and reminders

[1938] Server (generation) and device (adjustment): The server generates reminders based on schedule data and sends them to the device. The device adjusts the content and timing of the reminders based on the recognized emotions.

[1939] Input: Schedule data, emotional state.

[1940] Data calculation: Analyze schedule data and adjust reminders based on emotional state.

[1941] Output: The adjusted reminder is displayed on the display device.

[1942] What it does: If the user is in a high-stress state, the reminder is subtly displayed as "It's time for your next task. Please be mindful."

[1943] Step 6:

[1944] Activity monitoring and alerts

[1945] Terminal: Monitors the user's work activity and displays a warning if it differs from the scheduled data.

[1946] Input: Current activity, schedule data.

[1947] Data calculation: Compares activity content with schedule data and generates a warning if there is a discrepancy.

[1948] Output: A warning message is displayed.

[1949] What it does: If a user is browsing a website other than the scheduled task, a message will pop up asking, "Are you still working as planned?"

[1950] Prompt Sentence Examples

[1951] "If the user is not performing the current task accurately, display 'Are you continuing to work as planned?' and generate a prompt sentence recommending a break based on the emotion recognition results."

[1952] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1953] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1954] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1955] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1956] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1957] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1958] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1959] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1960] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1961] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1962] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1963] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1964] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1965] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1966] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1967] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1968] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1969] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1970] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1971] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1972] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1973] The following is further disclosed regarding the above embodiment.

[1974] (Claim 1)

[1975] A means for obtaining schedule data of a user;

[1976] A means for identifying required materials based on the acquired schedule data;

[1977] A means for automatically displaying the identified materials on the desktop of the user's terminal;

[1978] A system including:

[1979] (Claim 2)

[1980] means for monitoring the user's work activity;

[1981] a means for displaying an alert when the monitored activity differs from the schedule data;

[1982] 10. The system of claim 1, comprising:

[1983] (Claim 3)

[1984] means for analyzing user schedule data;

[1985] means for generating reminders based on the schedule data;

[1986] means for displaying the generated reminder on the user's terminal;

[1987] 10. The system of claim 1, comprising:

[1988] "Example 1"

[1989] (Claim 1)

[1990] A means for acquiring user schedule data from an external service;

[1991] A means for storing the acquired schedule data in a database;

[1992] a means of periodically updating the schedule data to keep it current;

[1993] A means for identifying necessary materials based on schedule data and automatically displaying them on the desktop based on the current time;

[1994] a means for monitoring the user's work activities and displaying a warning when the user is working on a task that differs from the schedule;

[1995] A way to analyze schedule data and generate reminders when important appointments or deadlines are approaching;

[1996] A means to display the generated reminders on your desktop;

[1997] A system including:

[1998] (Claim 2)

[1999] When a user logs in to the Desktop Magician system, a means for acquiring the user's latest schedule data and transmitting it to the terminal;

[2000] a means for displaying schedule-related materials on a desktop based on the received data;

[2001] 10. The system of claim 1, comprising:

[2002] (Claim 3)

[2003] a means for monitoring the title bar of the user's active window and displaying a warning popup if the title bar is different from what was expected;

[2004] The popup should inform the user that they are working on something other than what they had planned, and prompt them to return to their plan.

[2005] 10. The system of claim 1, comprising:

[2006] "Application Example 1"

[2007] (Claim 1)

[2008] A means for obtaining schedule data of a user;

[2009] A means for identifying required materials based on the acquired schedule data;

[2010] A means for automatically displaying the identified materials on the desktop of the user's terminal;

[2011] A means for factory robots to automatically prepare materials and tools,

[2012] A system including:

[2013] (Claim 2)

[2014] means for monitoring the user's work activity;

[2015] a means for displaying an alert when the monitored activity differs from the schedule data;

[2016] 10. The system of claim 1, comprising:

[2017] (Claim 3)

[2018] means for analyzing user schedule data;

[2019] means for generating reminders based on the schedule data;

[2020] means for displaying the generated reminder on the user's terminal;

[2021] A means of monitoring the progress of factory robots and providing efficient work instructions;

[2022] 10. The system of claim 1, comprising:

[2023] "Example 2: Combining Emotion Engines"

[2024] (Claim 1)

[2025] A means for obtaining schedule data of a user;

[2026] A means for identifying required materials based on the acquired schedule data;

[2027] A means for automatically displaying the identified materials on the desktop of the user's terminal;

[2028] means for recognizing the emotional state of a user;

[2029] means for adjusting the desktop display content based on the recognized emotional state;

[2030] A system including:

[2031] (Claim 2)

[2032] means for monitoring the user's work activity;

[2033] a means for displaying an alert when the monitored activity differs from the schedule data;

[2034] means for adjusting the alert content if the user's emotional state is excessively stressed;

[2035] 10. The system of claim 1, comprising:

[2036] (Claim 3)

[2037] means for analyzing user schedule data;

[2038] means for generating reminders based on the schedule data;

[2039] A means for adjusting the content and notification timing of the generated reminder based on the emotional state of the user;

[2040] 10. The system of claim 1, comprising:

[2041] "Application example 2 when combining emotion engines"

[2042] (Claim 1)

[2043] A means for obtaining schedule data of a user;

[2044] A means for identifying required materials based on the acquired schedule data;

[2045] means for automatically displaying the identified material on a display device on the user's terminal;

[2046] means for recognizing a user's emotion;

[2047] means for adjusting the displayed content based on the recognized emotion;

[2048] A system including:

[2049] (Claim 2)

[2050] means for monitoring the user's work activity;

[2051] a means for displaying an alert when the monitored activity differs from the schedule data;

[2052] a means for tailoring warning content based on emotion;

[2053] 10. The system of claim 1, comprising:

[2054] (Claim 3)

[2055] means for analyzing user schedule data;

[2056] means for generating reminders based on the schedule data;

[2057] means for displaying the generated reminder on the user's terminal;

[2058] means for adjusting the content and timing of reminders based on the perceived emotions;

[2059] 10. The system of claim 1, comprising: [Explanation of symbols]

[2060] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for obtaining schedule data of a user; A means for identifying required materials based on the acquired schedule data; A means for automatically displaying the identified materials on the desktop of the user's terminal; A system including:

2. means for monitoring the user's work activity; a means for displaying an alert when the monitored activity differs from the schedule data; The system of claim 1 , comprising:

3. means for analyzing user schedule data; means for generating reminders based on the schedule data; means for displaying the generated reminder on the user's terminal; The system of claim 1 , comprising:

Images