System

Abstract

A system is provided.SOLUTION: An operation terminal operated by a user, a server that processes input information received from the operation terminal, a database stored in the server, an interface that is displayed on the operation terminal and indicates an event or a task, and means for adjusting a schedule or detailing a task by a tap and release operation on the interface are included, and the server includes means for verifying authentication information received from the operation terminal, generating an authentication token, and transmitting the authentication token to the operation terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, interactions with artificial intelligence (AI) are often limited to text, voice input, and photos, which can be unintuitive for a wide variety of users. Furthermore, even everyday tasks such as scheduling and task detailing require complex procedures, placing a burden on users. This leaves many users unable to fully enjoy the convenience of AI. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides a system that includes an operation terminal operated by a user, a server that processes input information received from the operation terminal, a database stored on the server, an interface that displays events and tasks on the operation terminal, and means for adjusting schedules and detailing tasks by tapping and releasing on the interface.This system allows users to interact with AI simply and enjoyably through intuitive tapping and releasing, allowing them to perform everyday tasks efficiently.

[0006] An "operation terminal" refers to an electronic device that a user directly operates to input or check information.

[0007] "Server" refers to a computer system that receives input information sent from an operating terminal and processes and stores the data.

[0008] A "database" refers to a collection of data stored on a server that allows for efficient searching and management of data.

[0009] "Interface" refers to the visual elements and functions that are displayed on the screen of an operating terminal and allow users to interact with them.

[0010] A "tap and release operation" refers to an action in which a user taps a specific location on the touch screen of an operating device and then releases their finger.

[0011] "Event information" refers to information related to scheduling and task elaboration, and refers to data that indicates a specific activity or task.

[0012] "Time slot information" refers to data that indicates a specific time period during which an event or task is scheduled.

[0013] An "authentication token" refers to digital authentication information sent from a server to an operating terminal when a user is successfully authenticated. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The purpose of this invention is to utilize an interface that users can operate intuitively to more effectively and easily utilize the functions of artificial intelligence (AI). A specific embodiment of this system and its operation will be described below.

[0036] System Configuration

[0037] The system mainly consists of the following components:

[0038] 1. Operation terminal

[0039] A device that is directly operated by the user. Examples include smartphones and tablet devices.

[0040] It has a touch screen that displays the interface.

[0041] 2. Server

[0042] It receives input information sent from the operating terminal, processes it, and stores it.

[0043] Access the database and retrieve or register the required information.

[0044] 3. Database

[0045] It is stored on the server and stores schedule information, task information, etc.

[0046] 4. Interface

[0047] It is displayed on the operating device and is used by the user with a tap and release operation.

[0048] Events and tasks are visually displayed, allowing for intuitive operation.

[0049] Program processing

[0050] The following describes the program processing in this system. The processing of each function is described with the subject being either the operating terminal, the server, or the user.

[0051] Login process

[0052] 1. User: Launches the application and enters username and password on the login screen.

[0053] 2. Terminal: Sends the entered username and password to the server.

[0054] 3. Server: Validates the username and password against the database and, if authentication is successful, generates an authentication token.

[0055] 4. Server: Sends the authentication token to the operating terminal.

[0056] 5. Device: Stores the received authentication token locally and displays the user's home screen.

[0057] Schedule adjustment operations

[0058] 1. User: Tap the "Schedule" button from the home screen.

[0059] 2. Terminal: Displays the schedule adjustment screen and allows the user to select an event.

[0060] 3. User: Tap and release the event icon on the interface and drag it to the desired time frame.

[0061] 4. Terminal: Obtains the information of the selected event and time frame and sends it to the server.

[0062] 5. Server: Registers the received information in a database and sends the processing results to the terminal.

[0063] 6. Terminal: Receives the processing results, updates the schedule screen, and displays the registered details.

[0064] Task Detailing

[0065] 1. User: Tap the "Task Management" button on the schedule adjustment screen.

[0066] 2. Terminal: Displays the task management screen and provides text fields for entering details.

[0067] 3. User: Enter task details in the text field and tap the "Done" button.

[0068] 4. Terminal: Sends the entered task information to the server.

[0069] 5. Server: Registers the received information in a database and sends the processing results to the terminal.

[0070] 6. Terminal: Receives the processing results and updates the task management screen to display the new task.

[0071] Specific examples

[0072] For example, consider a user who wants to schedule a meeting from 9:00 to 10:00. The user first launches the application and enters their login information. After logging in, they navigate to the home screen and tap the "Schedule" button. The schedule adjustment screen appears, and the user drags the meeting icon to the 9:00 to 10:00 time slot. This operation causes the device to send the event information and time slot information to the server, which then registers it in the database. As a result, the end user can see that the schedule has been updated appropriately.

[0073] The above describes an embodiment of the present invention. This system allows users to intuitively utilize AI functions and efficiently adjust schedules and refine tasks.

[0074] The processing flow will be explained below.

[0075] Login process

[0076] Step 1:

[0077] User: Launches the application, enters the username and password on the login screen, and taps the "Login" button.

[0078] Step 2:

[0079] Terminal: Sends an authentication request to the server using the entered username and password.

[0080] Step 3:

[0081] Server: Based on the received request, the server checks the username and password in the database.

[0082] Step 4:

[0083] Server: If authentication is successful, generate an authentication token and send it to the terminal as a response.

[0084] Step 5:

[0085] Device: Receives the authentication token and stores it locally. After storing it, displays the home screen.

[0086] Schedule adjustment operations

[0087] Step 1:

[0088] User: Tap the Schedule button on the home screen.

[0089] Step 2:

[0090] Terminal: Display the schedule adjustment screen.

[0091] Step 3:

[0092] User: Tap and release the event icon (e.g., meeting icon) on the screen and drag it to the desired time slot (e.g., 9:00-10:00).

[0093] Step 4:

[0094] Terminal: Detects the user's tap and release operation and obtains the selected event information and time frame information.

[0095] Step 5:

[0096] Terminal: Creates and sends a request to send the acquired information to the server.

[0097] Step 6:

[0098] Server: Analyzes the received information and registers it in a database.

[0099] Step 7:

[0100] Server: If registration in the database is successful, it sends a success message to the terminal.

[0101] Step 8:

[0102] Terminal: Receives the success message and updates the schedule screen to show the new schedule.

[0103] Task Detailing

[0104] Step 1:

[0105] User: Tap the "Task Management" button on the schedule adjustment screen.

[0106] Step 2:

[0107] Terminal: Display the task management screen.

[0108] Step 3:

[0109] User: Taps a text field to enter new task information.

[0110] Step 4:

[0111] User: Enter task details in the text field and tap the "Done" button.

[0112] Step 5:

[0113] Terminal: Sends the entered task details to the server.

[0114] Step 6:

[0115] Server: Analyzes the received task information and registers it in the database.

[0116] Step 7:

[0117] Server: If registration in the database is successful, it sends a success message to the terminal.

[0118] Step 8:

[0119] Terminal: Receives the success message and updates the task management screen to show the new task information.

[0120] Example 1

[0121] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0122] In conventional systems, it was difficult for users to efficiently adjust schedules and detail tasks using an intuitive interface. The authentication process was also cumbersome, resulting in a poor user experience. Furthermore, there was a lack of a mechanism for reliably and securely transmitting task and event information to the server.

[0123] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0124] In this invention, the server includes means for processing input information received from the operation terminal, a database stored in the operation terminal, an interface means displayed on the operation terminal for displaying events and tasks, means for adjusting schedules and refining tasks through tap-and-release operations on the interface, means for verifying authentication information received from the operation terminal, generating an authentication token and transmitting it to the operation terminal, and means for transmitting event information and time slot information acquired based on the tap-and-release operations to the server. This allows users to effectively utilize AI functions through intuitive operations, enabling efficient schedule adjustments and task refinement.

[0125] An "information processing terminal" is a device that is operated by a user to input and display information.

[0126] The "arithmetic unit" is a device that processes input information received from an information processing terminal, and performs database management and various other operations.

[0127] An "information storage device" is a storage medium connected to a computing device for storing and managing data.

[0128] The "display means" is an interface that is built into the information processing terminal and that visually shows the work contents and schedule to the user.

[0129] The term "instruction operation" refers to an operation performed with a finger, a stylus, or the like using the display means of the information processing terminal.

[0130] "Authentication information" refers to information such as a user name and password that a user enters when using an information processing terminal.

[0131] An "authentication token" is data that indicates the authentication of a user and is generated when authentication information is successfully verified.

[0132] "Work information" is detailed information about tasks and events.

[0133] "Time information" is information about the time period in which a task or event is executed.

[0134] This invention relates to a system for effectively and easily adjusting schedules and detailing tasks via an intuitive interface on an information processing terminal (e.g., a smartphone or tablet) operated by a user.

[0135] System Configuration

[0136] The system mainly consists of the following components:

[0137] 1. Information processing terminal

[0138] It is a device that is directly operated by the user.

[0139] Examples include smartphones and tablet devices, which have touchscreens.

[0140] This terminal is equipped with a display means for visually showing the work content and schedule.

[0141] 2. Arithmetic device

[0142] It processes input information received from the information processing terminal and performs database management and various operations.

[0143] This applies to web servers or cloud servers.

[0144] 3. Information storage device

[0145] It is a storage medium connected to a computing device for storing and managing data.

[0146] A database management system (RDBMS) is used.

[0147] System Functions and Operation

[0148] Examples of the main functions of this system include login processing, schedule adjustment, and task detailing. The operation of each function is as follows:

[0149] Login process

[0150] 1. The user launches the application and enters authentication information (username and password) on the login screen.

[0151] 2. The information processing terminal transmits the input authentication information to the arithmetic unit.

[0152] 3. The computing device checks the authentication information against information stored in a database and, if successful, generates an authentication token.

[0153] 4. The computing device transmits the authentication token to the information processing terminal.

[0154] 5. The information processing terminal stores the authentication token in local storage and displays the user's home screen.

[0155] Schedule adjustment

[0156] 1. The user taps the "Schedule" button on the home screen.

[0157] 2. The information processing terminal displays the schedule screen, and the user drags and drops the schedule into the desired time slot.

[0158] 3. The information processing terminal transmits information about the selected schedule and time slot to the computing device.

[0159] 4. The arithmetic unit registers the received information in a database and returns the processing results to the information processing terminal.

[0160] 5. The information processing terminal receives the processing results and updates the schedule screen.

[0161] Task Detailing

[0162] 1. The user taps the "Task Management" button on the schedule adjustment screen.

[0163] 2. The information processing terminal displays the task management screen, and the user enters the task details.

[0164] 3. The information processing terminal transmits the input task information to the arithmetic unit.

[0165] 4. The arithmetic unit registers the received information in a database and returns the processing results to the information processing terminal.

[0166] 5. The information processing terminal receives the processing results and updates the task management screen.

[0167] Specific examples

[0168] For example, if a user "reschedules a meeting from 9 to 10":

[0169] 1. The user launches the application and enters their username and password at the login screen.

[0170] 2. The user taps the Schedule button on the home screen.

[0171] 3. The information processing terminal displays the schedule screen.

[0172] 4. The user drags and drops the meeting icon into the 9:00 to 10:00 time slot.

[0173] 5. The information processing terminal transmits the information about the selected meeting and the new time slot information to the computing device.

[0174] 6. The arithmetic unit registers the received data in a database and returns the processing results to the information processing terminal.

[0175] 7. The information processing terminal receives the processing results and displays that the schedule has been appropriately updated.

[0176] Example prompts for generative AI models

[0177] "Generate a sentence that explains the steps a user takes to reschedule a meeting from 9:00 to 10:00. The steps include launching the application, logging in, displaying the schedule screen, selecting the meeting icon, moving to the time slot, sending the data to the server, registering it in the database, and checking for updates."

[0178] This system allows users to intuitively utilize AI functions to efficiently adjust schedules and detail tasks.

[0179] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0180] Login process

[0181] Step 1:

[0182] A user launches an application and is directed to a login screen, where they enter their authentication information (username and password).

[0183] Input: Username and Password

[0184] Output: Create login request

[0185] Step 2:

[0186] The device sends the entered username and password to the server using the HTTPS protocol, which prevents unauthorized interception of the data.

[0187] Input: Username and Password

[0188] Output: HTTPS request

[0189] Step 3:

[0190] The server checks the received username and password against the information stored in the database, using an SQL query to look up the user information and confirm a match.

[0191] Input: Username and password in the HTTPS request

[0192] Output: Authentication result (success or failure)

[0193] Step 4:

[0194] If the server is successful in authenticating, it generates an authentication token, such as a JWT (JSON Web Token), which is used to ensure that subsequent requests are valid.

[0195] Input: Authentication result (success)

[0196] Output: Authentication token

[0197] Step 5:

[0198] The server sends the generated authentication token to the terminal using the HTTPS protocol.

[0199] Input: Authentication Token

[0200] Output: HTTPS response

[0201] Step 6:

[0202] The device stores the received authentication token in local storage and prepares to display the user's home screen.

[0203] Input: HTTPS response containing an authentication token

[0204] Output: Home screen display

[0205] Schedule adjustment operations

[0206] Step 1:

[0207] The user taps the "Schedule" button from the home screen, which takes the user to the schedule adjustment screen.

[0208] Input: Home screen interaction

[0209] Output: Schedule screen request

[0210] Step 2:

[0211] The device displays a schedule adjustment screen, providing a calendar and time slot interface.

[0212] Input: Schedule screen request

[0213] Output: Schedule adjustment screen

[0214] Step 3:

[0215] Users can visually change the schedule by dragging and dropping event icons onto the desired time slots on the interface.

[0216] Input: Drag and drop of event icons

[0217] Output: New timeframe selection information

[0218] Step 4:

[0219] The device sends the selected event information and new time slot information to the server in JSON format, which is used for subsequent processing.

[0220] Input: New timeframe selection information

[0221] Output: JSON format data

[0222] Step 5:

[0223] The server parses the received JSON data and executes SQL queries to register the relationship between events and time frames in the database.

[0224] Input: JSON format data

[0225] Output: Registration results in the database

[0226] Step 6:

[0227] The server returns the processing result (success or failure) to the terminal along with a status code.

[0228] Input: Registration results to the database

[0229] Output: Response with status code

[0230] Step 7:

[0231] The device receives the status code and updates the schedule screen accordingly. If successful, the new schedule information is displayed.

[0232] Input: Response with status code

[0233] Output: Updated schedule screen

[0234] Task Detailing

[0235] Step 1:

[0236] The user taps the "Task Management" button on the schedule adjustment screen, which takes the user to the task management screen.

[0237] Input: Schedule adjustment screen interaction

[0238] Output: Task management screen request

[0239] Step 2:

[0240] The device displays a task management screen, providing text fields and options for the user to enter task details.

[0241] Input: Task management screen request

[0242] Output: Task management screen

[0243] Step 3:

[0244] The user fills in the task details in the input fields and taps the "Done" button, which creates a new task.

[0245] Input: Enter task details

[0246] Output: New task information

[0247] Step 4:

[0248] The device sends new task information in JSON format to the server.

[0249] Input: New task information

[0250] Output: JSON format data

[0251] Step 5:

[0252] The server parses the received JSON data and executes an SQL query to register new task information in the database.

[0253] Input: JSON format data

[0254] Output: Registration results in the database

[0255] Step 6:

[0256] The server returns the processing result (success or failure) to the terminal along with a status code.

[0257] Input: Registration results to the database

[0258] Output: Response with status code

[0259] Step 7:

[0260] The device receives the status code and updates the task management screen accordingly, displaying the new task information if successful.

[0261] Input: Response with status code

[0262] Output: Updated task management screen

[0263] (Application example 1)

[0264] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0265] Managing robot operation schedules and task allocation within a factory is complex and time-consuming, placing a significant burden on managers. This burden increases even more when real-time adjustments and changes are required. Current systems lack an intuitive interface for performing these operations, making efficient management difficult. Therefore, there is a need for a method for efficiently managing robot operation schedules and task allocation within a factory using an intuitive interface.

[0266] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0267] In this invention, the server includes an operation terminal operated by a user, means for processing input information received from the operation terminal, a database stored in the server, an interface that indicates events and tasks and is displayed on the operation terminal, means for adjusting schedules and refining tasks by tapping and releasing on the interface, and means for intuitively managing robot operation schedules and task allocation within a factory via the operation terminal. This enables a user to efficiently manage robot operation schedules and task allocation within a factory using an interface that can be intuitively operated.

[0268] A "user" is a person who operates the system using an operation terminal.

[0269] An "operation terminal" is a device used by a user to adjust schedules and manage tasks, and includes smartphones, tablet terminals, and the like.

[0270] A "server" is a device that processes input information sent from an operating terminal and manages data stored in a database.

[0271] A "database" is a system that is stored on a server and stores data such as schedule information and task information.

[0272] An "interface" is a screen that is displayed on an operation terminal and allows a user to visually check events and tasks and perform operations.

[0273] A "tap and release operation" is a series of operations in which a user taps, drags, and then releases their finger on the touch screen of the operating terminal.

[0274] "Schedule adjustment" refers to an operation in which a user sets or changes the time frame of an event or task on the interface.

[0275] "Task elaboration" is an operation in which a user inputs and manages detailed information about a specific task.

[0276] A "robot" is an automated mechanical device used in factories that has the ability to perform various tasks.

[0277] A "robot operation schedule" is a plan that indicates what operations a robot will perform at what time periods.

[0278] "Task allocation" means assigning specific tasks to each robot.

[0279] This invention provides a system that allows intuitive management of robot operation schedules and task allocation within a factory. The main elements of the system are an operation terminal, a server, a database, and an interface. Specific mechanisms include login processing, schedule adjustment, and task detailing.

[0280] System Configuration

[0281] The system consists of the following elements:

[0282] 1. Operation terminal:

[0283] A device operated by a user, such as a smartphone or tablet.

[0284] It is equipped with a touch screen that allows users to operate it intuitively.

[0285] 2. Server:

[0286] This is a device that receives, processes, and saves input information sent from an operating terminal.

[0287] Access a database to manage schedule and task-related information.

[0288] 3. Database:

[0289] It is a system that stores schedule information, task information, etc. on a server.

[0290] 4. Interface:

[0291] This screen is displayed on the operation terminal and allows the user to visually check and operate events and tasks.

[0292] You can adjust schedules and detail tasks with a tap and release operation.

[0293] Program processing

[0294] Hardware and Software

[0295] Hardware:

[0296] Smartphone: A device running an operating system such as iOS or Android.

[0297] Server: A device that processes data and manages databases.

[0298] software:

[0299] Flask: A Python web framework for running server-side programs.

[0300] SQLite: A lightweight relational database management system.

[0301] Data processing and calculation

[0302] The server processes the authentication information received from the operation terminal and generates an authentication token. It also processes the event information and time slot information received from the operation terminal and registers it in a database. Specifically, when a user adjusts a schedule or details a task on the operation terminal, the operation terminal sends this data to the server. The server processes the received data, stores it in the database, and returns the results to the operation terminal. This allows users to intuitively manage the operation schedules and task allocation of robots in the factory.

[0303] Specific examples

[0304] For example, if a factory manager sets the operation schedule for Robot A from 1:00 to 2:00 and adds the task "inspection work," the process is as follows:

[0305] 1. Launch the app and log in by entering your username and password.

[0306] 2. Tap the "Schedule" button on the home screen and drag the Robot A icon to the 1:00 to 2:00 time slot.

[0307] 3. Next, tap the "Task Management" button, enter "Inspection Work," and tap "Done."

[0308] Prompt Sentence Examples

[0309] An example prompt is:

[0310] "Please generate a program to schedule Robot A from 1:00 to 2:00 and add 'Inspection Work' to the task."

[0311] This allows users to easily manage schedules and tasks for robots in their factories, improving factory operational efficiency.

[0312] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0313] Step 1:

[0314] The user starts up the operation terminal and enters the user name and password on the login screen. This input information is sent from the operation terminal to the server.

[0315] Input: Username, Password

[0316] Output: Authentication token, login success / failure message

[0317] Specific actions: The user uses the touchscreen to enter login information and taps the submit button.

[0318] Step 2:

[0319] The server verifies the received username and password in the database. If authentication is successful, it generates an authentication token and sends it to the operating terminal.

[0320] Input: Username, Password

[0321] Output: Authentication token

[0322] Specific operation: The server queries the database to verify the user information. If authentication is successful, the server generates an authentication token and sends it to the operating terminal.

[0323] Step 3:

[0324] The operation terminal locally stores the received authentication token and displays the user's home screen.

[0325] Input: Authentication Token

[0326] Output: Home screen

[0327] Specific operation: The operating terminal stores the authentication token and displays the home screen.

[0328] Step 4:

[0329] The user taps the "Schedule" button on the home screen to display the schedule adjustment screen.

[0330] Input: Tap operation

[0331] Output: Schedule adjustment screen

[0332] Specific actions: The user taps the "Schedule" icon on the home screen.

[0333] Step 5:

[0334] The user drags and drops a specific robot icon into the desired time slot on the schedule adjustment screen. The operating terminal acquires event information and time slot information based on this operation.

[0335] Input: Tap and release, drag

[0336] Output: Event information, time slot information

[0337] Specific behavior: The user touches the robot icon, drags it to the desired time frame, and releases it.

[0338] Step 6:

[0339] The operation terminal sends the acquired event information and time slot information to the server, which stores the received information in a database and updates the schedule based on that information.

[0340] Input: Event information, time slot information

[0341] Output: Schedule update information

[0342] Specific operation: The operation terminal sends the event information and time frame information to the server, and the server executes an update query on the database.

[0343] Step 7:

[0344] The server transmits the processing result to the operation terminal, and the operation terminal updates the schedule adjustment screen to display the new schedule.

[0345] Input: Schedule Update Information

[0346] Output: Updated schedule screen

[0347] Specific operation: The server sends schedule update information to the operation terminal, and the operation terminal updates the schedule screen.

[0348] Step 8:

[0349] The user taps the "Task Management" button on the schedule adjustment screen to display the task management screen.

[0350] Input: Tap operation

[0351] Output: Task management screen

[0352] Specific action: The user taps the "Task Management" button.

[0353] Step 9:

[0354] The user enters the details of the task to be assigned to a specific robot on the task management screen and taps the "Done" button. The operating device then sends this information to the server.

[0355] Input: Task details

[0356] Output: Task registration information

[0357] Specific actions: The user enters task details and taps the "Done" button.

[0358] Step 10:

[0359] The server stores the received task detail information in a database and transmits the processing results to the operating terminal.

[0360] Input: Task details

[0361] Output: Task registration result

[0362] Specific operation: The server saves the task details in the database and sends the processing results to the operating terminal.

[0363] Step 11:

[0364] The operation terminal updates the task management screen to display the new task.

[0365] Input: Task registration result

[0366] Output: Updated task management screen

[0367] Specific operation: The operation terminal updates the task management screen based on the task registration results and displays the new task.

[0368] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0369] System Configuration

[0370] This invention provides a system that combines a tap-and-release operation that allows users to perform operations intuitively with an emotion engine that recognizes the user's emotions. Specific embodiments of this system and their operation will be described below.

[0371] Basic Configuration

[0372] 1. Operation terminal:

[0373] These are devices that are directly operated by the user, such as smartphones and tablets.

[0374] It has a touch screen that displays the interface.

[0375] It also has a microphone and a camera for capturing the user's voice and facial expressions.

[0376] 2. Server:

[0377] It receives input information and emotion data sent from the operating terminal, processes them, and stores them.

[0378] Access the database and retrieve or register the required information.

[0379] 3. Database:

[0380] It is stored on the server and contains schedule information, task information, and authentication information.

[0381] 4. Interface:

[0382] It is displayed on the operating device and is used by the user with a tap and release operation.

[0383] Events and tasks are visually displayed, allowing for intuitive operation.

[0384] 5. Emotion Engine:

[0385] It analyzes the user's facial expressions, voice, or touch patterns to recognize emotions.

[0386] The interface display and operation methods are dynamically changed based on the recognized emotions.

[0387] Program processing

[0388] The program processing in this system will be explained below.

[0389] Login process

[0390] 1. User: Launch the application, enter your username and password on the login screen, and tap the "Login" button.

[0391] 2. Terminal: Sends an authentication request to the server using the entered username and password.

[0392] 3. Server: Based on the received request, it checks the username and password in the database.

[0393] 4. Server: If authentication is successful, generate an authentication token and send it to the terminal as a response.

[0394] 5. Device: Receives the authentication token and saves it locally. After saving, displays the home screen.

[0395] Schedule adjustment operations

[0396] 1. User: Tap the "Schedule" button on the home screen.

[0397] 2. Device: Display the schedule adjustment screen.

[0398] 3. User: Tap and release the event icon (e.g., meeting icon) on the screen and drag it to the desired time slot (e.g., 9:00-10:00).

[0399] 4. Terminal: Detects the user's tap and release operation and obtains the selected event information and time frame information.

[0400] 5. Terminal: Creates and sends a request to send the acquired information to the server.

[0401] 6. Server: Analyzes the received information and registers it in a database.

[0402] 7. Server: If registration in the database is successful, it sends a success message to the terminal.

[0403] 8. Terminal: Receives the success message and updates the schedule screen to show the new schedule.

[0404] Task Detailing

[0405] 1. User: Tap the "Task Management" button on the schedule adjustment screen.

[0406] 2. Device: Display the task management screen.

[0407] 3. User: Taps the text field to enter new task information.

[0408] 4. User: Enter task details in the text field and tap the "Done" button.

[0409] 5. Terminal: Sends the entered task details to the server.

[0410] 6. Server: Analyzes the received task information and registers it in the database.

[0411] 7. Server: If registration in the database is successful, it sends a success message to the terminal.

[0412] 8. Terminal: Receives the success message and updates the task management screen to show the new task information.

[0413] emotion recognition

[0414] 1. User: Facial expressions and voice are input by the emotion engine during operation.

[0415] 2. Terminal: The emotion engine analyzes the input data and recognizes the user's emotions.

[0416] 3. Terminal: Sends the recognized emotion data to the server.

[0417] 4. Server: Receives emotion data and sends instructions back to the device to dynamically adjust the interface display and operation method.

[0418] 5. Terminal: Dynamically changes the interface based on instructions from the server to adapt to the user's current emotional state.

[0419] Specific examples

[0420] For example, if the emotion engine detects that a user is feeling stressed while scheduling a meeting, that information is sent from the device to the server. The server can then make adjustments, such as simplifying the interface or displaying more relaxing colors and messages. This system significantly reduces the psychological burden on users during operation.

[0421] The above describes an embodiment of the present invention. This system allows users to intuitively utilize AI functions and efficiently manage tasks and schedules through an interface that adapts to their emotional state during operation.

[0422] The processing flow will be explained below.

[0423] Login process

[0424] Step 1:

[0425] User: Launches the application, enters the username and password on the login screen, and taps the "Login" button.

[0426] Step 2:

[0427] Terminal: Sends an authentication request to the server using the entered username and password.

[0428] Step 3:

[0429] Server: Based on the received request, the server checks the username and password in the database.

[0430] Step 4:

[0431] Server: If authentication is successful, generate an authentication token and send it to the terminal as a response.

[0432] Step 5:

[0433] Device: Receives the authentication token and stores it locally. After storing it, displays the home screen.

[0434] Schedule adjustment operations

[0435] Step 1:

[0436] User: Tap the Schedule button on the home screen.

[0437] Step 2:

[0438] Terminal: Display the schedule adjustment screen.

[0439] Step 3:

[0440] User: Tap and release the event icon (e.g., meeting icon) on the screen and drag it to the desired time slot (e.g., 9:00-10:00).

[0441] Step 4:

[0442] Terminal: Detects the user's tap and release operation and obtains the selected event information and time frame information.

[0443] Step 5:

[0444] Terminal: Creates and sends a request to send the acquired information to the server.

[0445] Step 6:

[0446] Server: Analyzes the received information and registers it in a database.

[0447] Step 7:

[0448] Server: If registration in the database is successful, it sends a success message to the terminal.

[0449] Step 8:

[0450] Terminal: Receives the success message and updates the schedule screen to show the new schedule.

[0451] Task Detailing

[0452] Step 1:

[0453] User: Tap the "Task Management" button on the schedule adjustment screen.

[0454] Step 2:

[0455] Terminal: Display the task management screen.

[0456] Step 3:

[0457] User: Taps a text field to enter new task information.

[0458] Step 4:

[0459] User: Enter task details in the text field and tap the "Done" button.

[0460] Step 5:

[0461] Terminal: Sends the entered task details to the server.

[0462] Step 6:

[0463] Server: Analyzes the received task information and registers it in the database.

[0464] Step 7:

[0465] Server: If registration in the database is successful, it sends a success message to the terminal.

[0466] Step 8:

[0467] Terminal: Receives the success message and updates the task management screen to show the new task information.

[0468] emotion recognition

[0469] Step 1:

[0470] User: During operation, emotions are input through the camera and microphone of the operating device.

[0471] Step 2:

[0472] Terminal: The emotion engine analyzes input data (facial expressions, voice, touch patterns) and recognizes the user's emotions.

[0473] Step 3:

[0474] Terminal: Sends the recognized emotion data to the server.

[0475] Step 4:

[0476] Server: Analyzes the received emotional data and generates instructions to dynamically change the interface display and operation method.

[0477] Step 5:

[0478] Server: Sends instructions to the terminal.

[0479] Step 6:

[0480] Terminal: Receives instructions from the server and dynamically changes the interface display and operation method.

[0481] Specific examples

[0482] For example, consider scheduling a meeting. A user launches an application, enters their login information, and logs in. From the home screen, they tap the "Schedule" button to move to the schedule adjustment screen. They drag the meeting icon to the 9:00 to 10:00 time slot, and the device sends this information to the server. The server registers the information in the database and sends a success message to the device. The device updates the schedule adjustment screen to show the meeting in the 9:00 to 10:00 time slot.

[0483] Furthermore, if the emotion engine detects stress from the user's facial expression during operation, it sends this information to the server. The server then generates instructions for relaxing color tones and message displays and sends them to the device. The device then dynamically changes its interface based on these instructions, improving the user's operating experience.

[0484] Example 2

[0485] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0486] Conventional schedule adjustment and task management systems using operating terminals often cause stress to users because they do not take into account the user's emotional state. Furthermore, when operations are complicated, users have difficulty efficiently managing tasks and schedules. Furthermore, there is a need for a method to smoothly access the system while ensuring the security of authentication information.

[0487] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0488] In this invention, the server includes an operation terminal operated by a user, means for processing input information received from the operation terminal, means for accessing a database stored on the server, means for adjusting schedules and detailing tasks by tapping and releasing on the interface, an emotion engine in the operation terminal that analyzes facial expressions and voice to recognize emotions, and means for transmitting the emotion data to the server and dynamically adjusting the display content and operation method of the interface. This enables dynamic adjustment of the interface according to the user's emotional state, providing a user-friendly operation environment and improving the efficiency of schedule adjustment and task management.

[0489] "User" refers to a person who operates the system via an operation terminal.

[0490] An "operation terminal" is a device operated by a user, and refers to devices such as smartphones and tablets.

[0491] "Server" refers to a computer system that processes input information received from an operating terminal and accesses a database.

[0492] "Database" refers to a data storage system that stores schedule information, task information, authentication information, etc.

[0493] "Interface" refers to a user interface that visually displays events and tasks on a screen displayed on an operating terminal.

[0494] "Tap and release operation" refers to an operation in which a user taps and releases their finger on the touchscreen of the operating device.

[0495] An "emotion engine" refers to a system that analyzes facial expressions and voices on an operating device to recognize emotions.

[0496] "Authentication token" refers to data generated by a server that contains authentication information for an authenticated user.

[0497] "Emotion data" refers to data that indicates the user's emotional state as recognized by the emotion engine.

[0498] System Configuration

[0499] This invention provides a system that combines a tap-and-release operation that allows users to perform operations intuitively with an emotion engine that recognizes the user's emotions. The basic configuration of the system is as follows.

[0500] Operation terminal

[0501] An operating terminal is a device that is directly operated by a user, such as a smartphone or tablet. Examples include iPhones and Android devices. These terminals are equipped with a touch screen, microphone, and camera, and display a user interface.

[0502] server

[0503] The server is a computer system that receives, processes, and stores input information and emotion data sent from the operating device. The server accesses a database to retrieve or register the necessary information. As a specific example, Django is used as the server-side framework.

[0504] Database

[0505] The database is a data storage system stored on a server for storing schedule information, task information, and authentication information. Specific examples include SQLite and MySQL.

[0506] Interface

[0507] The interface is displayed on the operating device and is used by the user with tap and release operations, visually indicating events and tasks and allowing intuitive operation.

[0508] Emotion Engine

[0509] An emotion engine is a system that recognizes emotions by analyzing a user's facial expressions, voice, or touch patterns. For example, OpenCV is used for facial recognition, and TensorFlow is used for emotion analysis. The interface display and operation methods are dynamically changed based on the recognized emotions.

[0510] Usage example

[0511] As a specific example, if the emotion engine detects that a user is feeling stressed while scheduling a meeting, that information is sent from the device to the server. The server can then make adjustments, such as simplifying the interface or displaying more relaxing colors and messages. This system significantly reduces the psychological burden on users during operation.

[0512] Example prompts to input to the generative AI model:

[0513] "How can I recognize a user's emotions while using a scheduling app and adjust the interface when the user feels stressed?"

[0514] The above description concretely illustrates the embodiments of the present invention. This system allows users to intuitively utilize AI functions and efficiently manage tasks and schedules through an interface that adapts to the user's emotional state during operation.

[0515] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0516] Login process

[0517] Step 1:

[0518] User: Launches the application, enters the username and password on the login screen, and taps the "Login" button.

[0519] Input: Username and Password

[0520] Output: Login operation trigger

[0521] Step 2:

[0522] Terminal: Sends an authentication request to the server using the entered username and password.

[0523] Specifically, the terminal generates an HTTP POST request and sends it to the server's authentication API endpoint.

[0524] Input: Username and Password

[0525] Output: Authentication request

[0526] Step 3:

[0527] Server: Based on the received request, the server checks the username and password in the database.

[0528] Specifically, the server runs a database query to verify that the username and hashed password match.

[0529] Input: Authentication Request

[0530] Output: Authentication result

[0531] Step 4:

[0532] Server: If authentication is successful, generate an authentication token and send it to the terminal as a response.

[0533] Specifically, it generates a JWT token along with a success message and returns it as an HTTP response.

[0534] Input: Authentication result

[0535] Output: Authentication token

[0536] Step 5:

[0537] Device: Receives the authentication token and stores it locally. After storing it, displays the home screen.

[0538] Specifically, the token is saved in local storage and the screen transitions to the next screen.

[0539] Input: Authentication Token

[0540] Output: Home screen display

[0541] Schedule adjustment operations

[0542] Step 1:

[0543] User: Tap the Schedule button on the home screen.

[0544] Input: User taps

[0545] Output: Display of schedule screen

[0546] Step 2:

[0547] Terminal: Display the schedule adjustment screen.

[0548] Input: Schedule screen display request

[0549] Output: Schedule adjustment screen

[0550] Step 3:

[0551] User: Tap and release the event icon (e.g., meeting icon) on the screen and drag it to the desired time slot (e.g., 9:00-10:00).

[0552] Input: Tap and release

[0553] Output: Drag operation completed

[0554] Step 4:

[0555] Terminal: Detects the user's tap and release operation and obtains the selected event information and time frame information.

[0556] Specifically, the event ID and time frame are stored as an internal data structure.

[0557] Input: Drag operation complete

[0558] Output: Event information and time slot information

[0559] Step 5:

[0560] Terminal: Creates and sends a request to send the acquired information to the server.

[0561] Specifically, it generates an HTTP POST request and sends it to the server's schedule API endpoint.

[0562] Input: Event information and time slot information

[0563] Output: Reschedule request

[0564] Step 6:

[0565] Server: Analyzes the received information and registers it in a database.

[0566] The specific operation is to execute an INSERT or UPDATE query on the database.

[0567] Input: Schedule change request

[0568] Output: Database update results

[0569] Step 7:

[0570] Server: If registration in the database is successful, it sends a success message to the terminal.

[0571] Specifically, a success status is returned in the HTTP response.

[0572] Input: Database update results

[0573] Output: Success message

[0574] Step 8:

[0575] Terminal: Receives the success message and updates the schedule screen to show the new schedule.

[0576] Specific operations include updating local data and redrawing the user interface.

[0577] Input: Success message

[0578] Output: Updated schedule contents

[0579] Task Detailing

[0580] Step 1:

[0581] User: Tap the "Task Management" button on the schedule adjustment screen.

[0582] Input: User taps

[0583] Output: Display of task management screen

[0584] Step 2:

[0585] Terminal: Display the task management screen.

[0586] Input: Request to display the task management screen

[0587] Output: Task management screen

[0588] Step 3:

[0589] User: Taps a text field to enter new task information.

[0590] Input: Tap operation

[0591] Output: Focus on text field

[0592] Step 4:

[0593] User: Enter task details in the text field and tap the "Done" button.

[0594] Input: Task information

[0595] Output: Task information input completed

[0596] Step 5:

[0597] Terminal: Sends the entered task details to the server.

[0598] Specifically, it generates an HTTP POST request and sends it to the server's task management API endpoint.

[0599] Input: Completed entering task information

[0600] Output: Task add request

[0601] Step 6:

[0602] Server: Analyzes the received task information and registers it in the database.

[0603] As a specific operation, an INSERT query is executed on the database.

[0604] Input: Task Add Request

[0605] Output: Database update results

[0606] Step 7:

[0607] Server: If registration in the database is successful, it sends a success message to the terminal.

[0608] Specifically, a success status is returned in the HTTP response.

[0609] Input: Database update results

[0610] Output: Success message

[0611] Step 8:

[0612] Terminal: Receives the success message and updates the task management screen to show the new task information.

[0613] Specific operations include updating local data and redrawing the user interface.

[0614] Input: Success message

[0615] Output: Updated task information

[0616] emotion recognition

[0617] Step 1:

[0618] User: Facial expressions and voice are input by the emotion engine during operation.

[0619] Input: Facial expressions and voice data

[0620] Output: Input to the emotion engine

[0621] Step 2:

[0622] Terminal: The emotion engine analyzes the input data and recognizes the user's emotions.

[0623] Specifically, it uses machine learning models to classify emotions.

[0624] Input: Facial expressions and voice data

[0625] Output: Recognized emotion data

[0626] Step 3:

[0627] Terminal: Sends the recognized emotion data to the server.

[0628] Specifically, it creates emotion data in JSON format and generates and sends an HTTP POST request.

[0629] Input: Recognized emotion data

[0630] Output: Sending emotion data

[0631] Step 4:

[0632] Server: Receives emotion data and sends instructions back to the device to dynamically adjust the interface display and operation method.

[0633] Specifically, it generates UI change instructions according to the emotion and returns them as an HTTP response.

[0634] Input: Emotion data

[0635] Output: UI change instructions

[0636] Step 5:

[0637] Terminal: Receives instructions from the server and dynamically changes the interface to adapt to the user's current emotional state.

[0638] Specific actions include changing the layout and color of the interface.

[0639] Input: UI change instructions

[0640] Output: Dynamically changed interface

[0641] (Application example 2)

[0642] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0643] Conventional systems lack an operational interface that takes user emotions into account, which means that operability declines when users feel stressed. Furthermore, there is a lack of systems to improve individual customer experiences in brick-and-mortar stores, and there is also the problem that customization suggestions based on user emotions have not yet been fully implemented.

[0644] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0645] In this invention, the server includes an operation terminal operated by a user, a central processing unit that processes input information received from the operation terminal, an information storage device that stores information stored in the central processing unit, a display unit that displays tasks on the operation terminal, a unit that adjusts schedules and subdivides tasks through selection operations on the display unit, an emotion recognition unit that analyzes the user's facial expression and voice data, and a unit that dynamically changes the content of the display unit based on the emotion identified by the emotion recognition unit. This makes it possible to dynamically adjust the interface according to the user's emotion and improve the individual customer experience in physical stores.

[0646] An "operation terminal" is a device that is directly operated by a user, such as a smartphone or tablet.

[0647] The "central processing unit" is a device that processes input information received from an operation terminal, and is usually a computer system called a server.

[0648] "Information storage device" means a device for storing data saved in a central processing unit, and includes a database.

[0649] The "display means" is an interface for displaying tasks and events on the operating terminal, and is generally a touch screen display.

[0650] The "selection operation" refers to an action in which a user selects a task or event on the display means, and includes finger movements such as tapping and swiping.

[0651] "Emotion recognition means" is a technology that identifies emotions by analyzing a user's facial expressions and voice data, and machine learning models are often used.

[0652] The "dynamically changing means" refers to a function of adjusting the content of the display means in real time based on the emotion identified by the emotion recognition means.

[0653] The embodiment of the present invention is a system comprising an operation terminal operated by a user, a central processing unit, an information storage device, a display means, a selection operation means, an emotion recognition means, and a dynamic change means. A specific embodiment of this system will be described in detail below.

[0654] System Configuration

[0655] 1. Operation terminal: A device that is directly operated by the user, such as a smartphone or tablet. It is equipped with a touch screen, camera, and microphone, and can capture the user's facial expressions and voice.

[0656] 2. Central Processing Unit: A computer system that processes input information received from the operating terminal, usually called a server. It integrates multiple input information and retrieves and registers data from the information storage device.

[0657] 3. Information storage device: A device that includes a database and stores schedule information, task information, user authentication information, emotion data, etc.

[0658] 4. Display: An interface displayed on the operating device, providing a visual display to intuitively show tasks and events. It accepts operations such as tapping and swiping.

[0659] 5. Selection operation means: This function detects tap and release operations performed by the user on the display means and allows for schedule adjustments and task subdivision.

[0660] 6. Emotion recognition: Analyzes facial expressions and voice data acquired from the smartphone camera and microphone to identify the user's emotions. Uses machine learning models (e.g., TensorFlow, PyTorch).

[0661] 7. Dynamic modification: This function adjusts the content of the display means in real time based on the emotional data identified by the emotion recognition means. The interface is modified in a way that adapts to the user's current emotional state.

[0662] Program processing explanation

[0663] 1. User: Uses a smartphone to launch the application and operate the interface shown on the display means.

[0664] 2. Device: Detects selection operations such as taps and swipes performed by the user. Also, captures the user's facial expressions and voice using a camera and microphone. This data is sent to the emotion recognition system.

[0665] 3. Emotion recognition means: Analyzes facial expressions and voice data acquired using a machine learning model to identify the user's emotions. The identified emotion data is sent to the central processing unit.

[0666] 4. Central processing unit: After receiving the emotion data, it retrieves related data from the information storage device and generates content to be sent to the display means. It also generates product suggestions and schedule information according to the emotion.

[0667] 5. Display means: Based on the data sent from the central processing unit, the interface content is dynamically changed and displayed to the user, such as special product offers, event information, or relaxing color themes.

[0668] Specific examples

[0669] For example, if a user stops in front of a particular product shelf in a physical store and looks confused, the smartphone's camera and microphone will capture this. The emotion recognition means will identify the emotion as "confusion" and send that data to the central processing unit. The central processing unit will then generate an interface that provides the confused user with detailed product information and usage instructions, and display this on the display means in real time.

[0670] Prompt Sentence Examples

[0671] "Users' emotions are confusing. Can you recommend any products?"

[0672] This system allows the interface to dynamically adjust based on the user's emotions, significantly improving the personalized customer experience in physical stores.

[0673] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0674] Step 1:

[0675] The user starts the application using a smartphone, enters the username and password on the login screen, and taps the "Login" button.

[0676] Input: Username, Password

[0677] Output: Generated authentication request

[0678] Specifically, it detects input on the smartphone's touchscreen and creates an authentication request.

[0679] Step 2:

[0680] The device sends an authentication request to the server.

[0681] Input: Authentication request

[0682] Output: Send data to the server

[0683] Specifically, the network function of the smartphone is used to send authentication request data to the server.

[0684] Step 3:

[0685] Based on the request received by the server, the information storage device checks the user name and password.

[0686] Input: Authentication request (username, password)

[0687] Output: Authentication result (success or failure)

[0688] Specifically, it performs a database query and compares the received data with the stored data.

[0689] Step 4:

[0690] If the server is successful in the authentication, it generates an authentication token and sends it to the terminal as a response.

[0691] Input: Authentication result (success)

[0692] Output: Authentication token

[0693] Specifically, the authentication token is generated using a random number generation algorithm and sent to the terminal.

[0694] Step 5:

[0695] The device receives the authentication token, stores it locally, and displays the home screen.

[0696] Input: Authentication token

[0697] Output: Home screen display

[0698] Specifically, it stores the token in local storage and updates the application's UI component to display the home screen.

[0699] Step 6:

[0700] As the user walks around the store, the device uses a camera and microphone to capture the user's facial expressions and voice.

[0701] Input: Camera video, audio data

[0702] Output: Captured data

[0703] Specifically, it periodically collects data from the camera and microphone sensors.

[0704] Step 7:

[0705] An emotion recognition means in the terminal analyzes the captured data and identifies the user's emotion.

[0706] Input: Captured data (facial expression, voice)

[0707] Output: Recognized emotion data

[0708] Specifically, it uses machine learning models to analyze data and identify specific emotions.

[0709] Step 8:

[0710] The device transmits the recognized emotion data to the server.

[0711] Input: Emotion data

[0712] Output: Send emotion data to the server

[0713] As a specific operation, emotion data is transmitted via a network.

[0714] Step 9:

[0715] The server receives the emotion data and uses a generative AI model to generate data for appropriate product suggestions and interface changes.

[0716] Input: Emotion data

[0717] Output: Proposal data, interface change data

[0718] Specifically, the prompt sentence "The user is confused. Please recommend some products." is input into the AI ​​model, and the generated suggestions are obtained.

[0719] Step 10:

[0720] The server transmits the generated proposal data to the terminal.

[0721] Input: Proposal data

[0722] Output: Sending data to the terminal

[0723] As a specific operation, the proposal data is transmitted to the terminal via the network.

[0724] Step 11:

[0725] The terminal receives the proposal data and dynamically adjusts and presents an interface to the user using the display means.

[0726] Input: Proposal data

[0727] Output: Dynamically adjusted interface

[0728] Specifically, it updates interface elements such as display content, color themes, and messages and displays them to the user.

[0729] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0730] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0731] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0732] [Second embodiment]

[0733] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0734] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0735] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0736] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0737] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0738] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0739] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0740] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0741] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0742] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0743] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0744] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0745] The purpose of this invention is to utilize an interface that users can operate intuitively to more effectively and easily utilize the functions of artificial intelligence (AI). A specific embodiment of this system and its operation will be described below.

[0746] System Configuration

[0747] The system mainly consists of the following components:

[0748] 1. Operation terminal

[0749] A device that is directly operated by the user. Examples include smartphones and tablet devices.

[0750] It has a touch screen that displays the interface.

[0751] 2. Server

[0752] It receives input information sent from the operating terminal, processes it, and stores it.

[0753] Access the database and retrieve or register the required information.

[0754] 3. Database

[0755] It is stored on the server and stores schedule information, task information, etc.

[0756] 4. Interface

[0757] It is displayed on the operating device and is used by the user with a tap and release operation.

[0758] Events and tasks are visually displayed, allowing for intuitive operation.

[0759] Program processing

[0760] The following describes the program processing in this system. The processing of each function is described with the subject being either the operating terminal, the server, or the user.

[0761] Login process

[0762] 1. User: Launches the application and enters username and password on the login screen.

[0763] 2. Terminal: Sends the entered username and password to the server.

[0764] 3. Server: Validates the username and password against the database and, if authentication is successful, generates an authentication token.

[0765] 4. Server: Sends the authentication token to the operating terminal.

[0766] 5. Device: Stores the received authentication token locally and displays the user's home screen.

[0767] Schedule adjustment operations

[0768] 1. User: Tap the "Schedule" button from the home screen.

[0769] 2. Terminal: Displays the schedule adjustment screen and allows the user to select an event.

[0770] 3. User: Tap and release the event icon on the interface and drag it to the desired time frame.

[0771] 4. Terminal: Obtains the information of the selected event and time frame and sends it to the server.

[0772] 5. Server: Registers the received information in a database and sends the processing results to the terminal.

[0773] 6. Terminal: Receives the processing results, updates the schedule screen, and displays the registered details.

[0774] Task Detailing

[0775] 1. User: Tap the "Task Management" button on the schedule adjustment screen.

[0776] 2. Terminal: Displays the task management screen and provides text fields for entering details.

[0777] 3. User: Enter task details in the text field and tap the "Done" button.

[0778] 4. Terminal: Sends the entered task information to the server.

[0779] 5. Server: Registers the received information in a database and sends the processing results to the terminal.

[0780] 6. Terminal: Receives the processing results and updates the task management screen to display the new task.

[0781] Specific examples

[0782] For example, consider a user who wants to schedule a meeting from 9:00 to 10:00. The user first launches the application and enters their login information. After logging in, they navigate to the home screen and tap the "Schedule" button. The schedule adjustment screen appears, and the user drags the meeting icon to the 9:00 to 10:00 time slot. This operation causes the device to send the event information and time slot information to the server, which then registers it in the database. As a result, the end user can see that the schedule has been updated appropriately.

[0783] The above describes an embodiment of the present invention. This system allows users to intuitively utilize AI functions and efficiently adjust schedules and refine tasks.

[0784] The processing flow will be explained below.

[0785] Login process

[0786] Step 1:

[0787] User: Launches the application, enters the username and password on the login screen, and taps the "Login" button.

[0788] Step 2:

[0789] Terminal: Sends an authentication request to the server using the entered username and password.

[0790] Step 3:

[0791] Server: Based on the received request, the server checks the username and password in the database.

[0792] Step 4:

[0793] Server: If authentication is successful, generate an authentication token and send it to the terminal as a response.

[0794] Step 5:

[0795] Device: Receives the authentication token and stores it locally. After storing it, displays the home screen.

[0796] Schedule adjustment operations

[0797] Step 1:

[0798] User: Tap the Schedule button on the home screen.

[0799] Step 2:

[0800] Terminal: Display the schedule adjustment screen.

[0801] Step 3:

[0802] User: Tap and release the event icon (e.g., meeting icon) on the screen and drag it to the desired time slot (e.g., 9:00-10:00).

[0803] Step 4:

[0804] Terminal: Detects the user's tap and release operation and obtains the selected event information and time frame information.

[0805] Step 5:

[0806] Terminal: Creates and sends a request to send the acquired information to the server.

[0807] Step 6:

[0808] Server: Analyzes the received information and registers it in a database.

[0809] Step 7:

[0810] Server: If registration in the database is successful, it sends a success message to the terminal.

[0811] Step 8:

[0812] Terminal: Receives the success message and updates the schedule screen to show the new schedule.

[0813] Task Detailing

[0814] Step 1:

[0815] User: Tap the "Task Management" button on the schedule adjustment screen.

[0816] Step 2:

[0817] Terminal: Display the task management screen.

[0818] Step 3:

[0819] User: Taps a text field to enter new task information.

[0820] Step 4:

[0821] User: Enter task details in the text field and tap the "Done" button.

[0822] Step 5:

[0823] Terminal: Sends the entered task details to the server.

[0824] Step 6:

[0825] Server: Analyzes the received task information and registers it in the database.

[0826] Step 7:

[0827] Server: If registration in the database is successful, it sends a success message to the terminal.

[0828] Step 8:

[0829] Terminal: Receives the success message and updates the task management screen to show the new task information.

[0830] Example 1

[0831] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0832] In conventional systems, it was difficult for users to efficiently adjust schedules and detail tasks using an intuitive interface. The authentication process was also cumbersome, resulting in a poor user experience. Furthermore, there was a lack of a mechanism for reliably and securely transmitting task and event information to the server.

[0833] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0834] In this invention, the server includes means for processing input information received from the operation terminal, a database stored in the operation terminal, an interface means displayed on the operation terminal for displaying events and tasks, means for adjusting schedules and refining tasks through tap-and-release operations on the interface, means for verifying authentication information received from the operation terminal, generating an authentication token and transmitting it to the operation terminal, and means for transmitting event information and time slot information acquired based on the tap-and-release operations to the server. This allows users to effectively utilize AI functions through intuitive operations, enabling efficient schedule adjustments and task refinement.

[0835] An "information processing terminal" is a device that is operated by a user to input and display information.

[0836] The "arithmetic unit" is a device that processes input information received from an information processing terminal, and performs database management and various other operations.

[0837] An "information storage device" is a storage medium connected to a computing device for storing and managing data.

[0838] The "display means" is an interface that is built into the information processing terminal and that visually shows the work contents and schedule to the user.

[0839] The term "instruction operation" refers to an operation performed with a finger, a stylus, or the like using the display means of the information processing terminal.

[0840] "Authentication information" refers to information such as a user name and password that a user enters when using an information processing terminal.

[0841] An "authentication token" is data that indicates the authentication of a user and is generated when authentication information is successfully verified.

[0842] "Work information" is detailed information about tasks and events.

[0843] "Time information" is information about the time period in which a task or event is executed.

[0844] This invention relates to a system for effectively and easily adjusting schedules and detailing tasks via an intuitive interface on an information processing terminal (e.g., a smartphone or tablet) operated by a user.

[0845] System Configuration

[0846] The system mainly consists of the following components:

[0847] 1. Information processing terminal

[0848] It is a device that is directly operated by the user.

[0849] Examples include smartphones and tablet devices, which have touchscreens.

[0850] This terminal is equipped with a display means for visually showing the work content and schedule.

[0851] 2. Arithmetic device

[0852] It processes input information received from the information processing terminal and performs database management and various operations.

[0853] This applies to web servers or cloud servers.

[0854] 3. Information storage device

[0855] It is a storage medium connected to a computing device for storing and managing data.

[0856] A database management system (RDBMS) is used.

[0857] System Functions and Operation

[0858] Examples of the main functions of this system include login processing, schedule adjustment, and task detailing. The operation of each function is as follows:

[0859] Login process

[0860] 1. The user launches the application and enters authentication information (username and password) on the login screen.

[0861] 2. The information processing terminal transmits the input authentication information to the arithmetic unit.

[0862] 3. The computing device checks the authentication information against information stored in a database and, if successful, generates an authentication token.

[0863] 4. The computing device transmits the authentication token to the information processing terminal.

[0864] 5. The information processing terminal stores the authentication token in local storage and displays the user's home screen.

[0865] Schedule adjustment

[0866] 1. The user taps the "Schedule" button on the home screen.

[0867] 2. The information processing terminal displays the schedule screen, and the user drags and drops the schedule into the desired time slot.

[0868] 3. The information processing terminal transmits information about the selected schedule and time slot to the computing device.

[0869] 4. The arithmetic unit registers the received information in a database and returns the processing results to the information processing terminal.

[0870] 5. The information processing terminal receives the processing results and updates the schedule screen.

[0871] Task Detailing

[0872] 1. The user taps the "Task Management" button on the schedule adjustment screen.

[0873] 2. The information processing terminal displays the task management screen, and the user enters the task details.

[0874] 3. The information processing terminal transmits the input task information to the arithmetic unit.

[0875] 4. The arithmetic unit registers the received information in a database and returns the processing results to the information processing terminal.

[0876] 5. The information processing terminal receives the processing results and updates the task management screen.

[0877] Specific examples

[0878] For example, if a user "reschedules a meeting from 9 to 10":

[0879] 1. The user launches the application and enters their username and password at the login screen.

[0880] 2. The user taps the Schedule button on the home screen.

[0881] 3. The information processing terminal displays the schedule screen.

[0882] 4. The user drags and drops the meeting icon into the 9:00 to 10:00 time slot.

[0883] 5. The information processing terminal transmits the information about the selected meeting and the new time slot information to the computing device.

[0884] 6. The arithmetic unit registers the received data in a database and returns the processing results to the information processing terminal.

[0885] 7. The information processing terminal receives the processing results and displays that the schedule has been appropriately updated.

[0886] Example prompts for generative AI models

[0887] "Generate a sentence that explains the steps a user takes to reschedule a meeting from 9:00 to 10:00. The steps include launching the application, logging in, displaying the schedule screen, selecting the meeting icon, moving to the time slot, sending the data to the server, registering it in the database, and checking for updates."

[0888] This system allows users to intuitively utilize AI functions to efficiently adjust schedules and detail tasks.

[0889] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0890] Login process

[0891] Step 1:

[0892] A user launches an application and is directed to a login screen, where they enter their authentication information (username and password).

[0893] Input: Username and Password

[0894] Output: Create login request

[0895] Step 2:

[0896] The device sends the entered username and password to the server using the HTTPS protocol, which prevents unauthorized interception of the data.

[0897] Input: Username and Password

[0898] Output: HTTPS request

[0899] Step 3:

[0900] The server checks the received username and password against the information stored in the database, using an SQL query to look up the user information and confirm a match.

[0901] Input: Username and password in the HTTPS request

[0902] Output: Authentication result (success or failure)

[0903] Step 4:

[0904] If the server is successful in authenticating, it generates an authentication token, such as a JWT (JSON Web Token), which is used to ensure that subsequent requests are valid.

[0905] Input: Authentication result (success)

[0906] Output: Authentication token

[0907] Step 5:

[0908] The server sends the generated authentication token to the terminal using the HTTPS protocol.

[0909] Input: Authentication Token

[0910] Output: HTTPS response

[0911] Step 6:

[0912] The device stores the received authentication token in local storage and prepares to display the user's home screen.

[0913] Input: HTTPS response containing an authentication token

[0914] Output: Home screen display

[0915] Schedule adjustment operations

[0916] Step 1:

[0917] The user taps the "Schedule" button from the home screen, which takes the user to the schedule adjustment screen.

[0918] Input: Home screen interaction

[0919] Output: Schedule screen request

[0920] Step 2:

[0921] The device displays a schedule adjustment screen, providing a calendar and time slot interface.

[0922] Input: Schedule screen request

[0923] Output: Schedule adjustment screen

[0924] Step 3:

[0925] Users can visually change the schedule by dragging and dropping event icons onto the desired time slots on the interface.

[0926] Input: Drag and drop of event icons

[0927] Output: New timeframe selection information

[0928] Step 4:

[0929] The device sends the selected event information and new time slot information to the server in JSON format, which is used for subsequent processing.

[0930] Input: New timeframe selection information

[0931] Output: JSON format data

[0932] Step 5:

[0933] The server parses the received JSON data and executes SQL queries to register the relationship between events and time frames in the database.

[0934] Input: JSON format data

[0935] Output: Registration results in the database

[0936] Step 6:

[0937] The server returns the processing result (success or failure) to the terminal along with a status code.

[0938] Input: Registration results to the database

[0939] Output: Response with status code

[0940] Step 7:

[0941] The device receives the status code and updates the schedule screen accordingly. If successful, the new schedule information is displayed.

[0942] Input: Response with status code

[0943] Output: Updated schedule screen

[0944] Task Detailing

[0945] Step 1:

[0946] The user taps the "Task Management" button on the schedule adjustment screen, which takes the user to the task management screen.

[0947] Input: Schedule adjustment screen interaction

[0948] Output: Task management screen request

[0949] Step 2:

[0950] The device displays a task management screen, providing text fields and options for the user to enter task details.

[0951] Input: Task management screen request

[0952] Output: Task management screen

[0953] Step 3:

[0954] The user fills in the task details in the input fields and taps the "Done" button, which creates a new task.

[0955] Input: Enter task details

[0956] Output: New task information

[0957] Step 4:

[0958] The device sends new task information in JSON format to the server.

[0959] Input: New task information

[0960] Output: JSON format data

[0961] Step 5:

[0962] The server parses the received JSON data and executes an SQL query to register new task information in the database.

[0963] Input: JSON format data

[0964] Output: Registration results in the database

[0965] Step 6:

[0966] The server returns the processing result (success or failure) to the terminal along with a status code.

[0967] Input: Registration results to the database

[0968] Output: Response with status code

[0969] Step 7:

[0970] The device receives the status code and updates the task management screen accordingly, displaying the new task information if successful.

[0971] Input: Response with status code

[0972] Output: Updated task management screen

[0973] (Application example 1)

[0974] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0975] Managing robot operation schedules and task allocation within a factory is complex and time-consuming, placing a significant burden on managers. This burden increases even more when real-time adjustments and changes are required. Current systems lack an intuitive interface for performing these operations, making efficient management difficult. Therefore, there is a need for a method for efficiently managing robot operation schedules and task allocation within a factory using an intuitive interface.

[0976] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0977] In this invention, the server includes an operation terminal operated by a user, means for processing input information received from the operation terminal, a database stored in the server, an interface that indicates events and tasks and is displayed on the operation terminal, means for adjusting schedules and refining tasks by tapping and releasing on the interface, and means for intuitively managing robot operation schedules and task allocation within a factory via the operation terminal. This enables a user to efficiently manage robot operation schedules and task allocation within a factory using an interface that can be intuitively operated.

[0978] A "user" is a person who operates the system using an operation terminal.

[0979] An "operation terminal" is a device used by a user to adjust schedules and manage tasks, and includes smartphones, tablet terminals, and the like.

[0980] A "server" is a device that processes input information sent from an operating terminal and manages data stored in a database.

[0981] A "database" is a system that is stored on a server and stores data such as schedule information and task information.

[0982] An "interface" is a screen that is displayed on an operation terminal and allows a user to visually check events and tasks and perform operations.

[0983] A "tap and release operation" is a series of operations in which a user taps, drags, and then releases their finger on the touch screen of the operating terminal.

[0984] "Schedule adjustment" refers to an operation in which a user sets or changes the time frame of an event or task on the interface.

[0985] "Task elaboration" is an operation in which a user inputs and manages detailed information about a specific task.

[0986] A "robot" is an automated mechanical device used in factories that has the ability to perform various tasks.

[0987] A "robot operation schedule" is a plan that indicates what operations a robot will perform at what time periods.

[0988] "Task allocation" means assigning specific tasks to each robot.

[0989] This invention provides a system that allows intuitive management of robot operation schedules and task allocation within a factory. The main elements of the system are an operation terminal, a server, a database, and an interface. Specific mechanisms include login processing, schedule adjustment, and task detailing.

[0990] System Configuration

[0991] The system consists of the following elements:

[0992] 1. Operation terminal:

[0993] A device operated by a user, such as a smartphone or tablet.

[0994] It is equipped with a touch screen that allows users to operate it intuitively.

[0995] 2. Server:

[0996] This is a device that receives, processes, and saves input information sent from an operating terminal.

[0997] Access a database to manage schedule and task-related information.

[0998] 3. Database:

[0999] It is a system that stores schedule information, task information, etc. on a server.

[1000] 4. Interface:

[1001] This screen is displayed on the operation terminal and allows the user to visually check and operate events and tasks.

[1002] You can adjust schedules and detail tasks with a tap and release operation.

[1003] Program processing

[1004] Hardware and Software

[1005] Hardware:

[1006] Smartphone: A device running an operating system such as iOS or Android.

[1007] Server: A device that processes data and manages databases.

[1008] software:

[1009] Flask: A Python web framework for running server-side programs.

[1010] SQLite: A lightweight relational database management system.

[1011] Data processing and calculation

[1012] The server processes the authentication information received from the operation terminal and generates an authentication token. It also processes the event information and time slot information received from the operation terminal and registers it in a database. Specifically, when a user adjusts a schedule or details a task on the operation terminal, the operation terminal sends this data to the server. The server processes the received data, stores it in the database, and returns the results to the operation terminal. This allows users to intuitively manage the operation schedules and task allocation of robots in the factory.

[1013] Specific examples

[1014] For example, if a factory manager sets the operation schedule for Robot A from 1:00 to 2:00 and adds the task "inspection work," the process is as follows:

[1015] 1. Launch the app and log in by entering your username and password.

[1016] 2. Tap the "Schedule" button on the home screen and drag the Robot A icon to the 1:00 to 2:00 time slot.

[1017] 3. Next, tap the "Task Management" button, enter "Inspection Work," and tap "Done."

[1018] Prompt Sentence Examples

[1019] An example prompt is:

[1020] "Please generate a program to schedule Robot A from 1:00 to 2:00 and add 'Inspection Work' to the task."

[1021] This allows users to easily manage schedules and tasks for robots in their factories, improving factory operational efficiency.

[1022] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1023] Step 1:

[1024] The user starts up the operation terminal and enters the user name and password on the login screen. This input information is sent from the operation terminal to the server.

[1025] Input: Username, Password

[1026] Output: Authentication token, login success / failure message

[1027] Specific actions: The user uses the touchscreen to enter login information and taps the submit button.

[1028] Step 2:

[1029] The server verifies the received username and password in the database. If authentication is successful, it generates an authentication token and sends it to the operating terminal.

[1030] Input: Username, Password

[1031] Output: Authentication token

[1032] Specific operation: The server queries the database to verify the user information. If authentication is successful, the server generates an authentication token and sends it to the operating terminal.

[1033] Step 3:

[1034] The operation terminal locally stores the received authentication token and displays the user's home screen.

[1035] Input: Authentication Token

[1036] Output: Home screen

[1037] Specific operation: The operating terminal stores the authentication token and displays the home screen.

[1038] Step 4:

[1039] The user taps the "Schedule" button on the home screen to display the schedule adjustment screen.

[1040] Input: Tap operation

[1041] Output: Schedule adjustment screen

[1042] Specific actions: The user taps the "Schedule" icon on the home screen.

[1043] Step 5:

[1044] The user drags and drops a specific robot icon into the desired time slot on the schedule adjustment screen. The operating terminal acquires event information and time slot information based on this operation.

[1045] Input: Tap and release, drag

[1046] Output: Event information, time slot information

[1047] Specific behavior: The user touches the robot icon, drags it to the desired time frame, and releases it.

[1048] Step 6:

[1049] The operation terminal sends the acquired event information and time slot information to the server, which stores the received information in a database and updates the schedule based on that information.

[1050] Input: Event information, time slot information

[1051] Output: Schedule update information

[1052] Specific operation: The operation terminal sends the event information and time frame information to the server, and the server executes an update query on the database.

[1053] Step 7:

[1054] The server transmits the processing result to the operation terminal, and the operation terminal updates the schedule adjustment screen to display the new schedule.

[1055] Input: Schedule Update Information

[1056] Output: Updated schedule screen

[1057] Specific operation: The server sends schedule update information to the operation terminal, and the operation terminal updates the schedule screen.

[1058] Step 8:

[1059] The user taps the "Task Management" button on the schedule adjustment screen to display the task management screen.

[1060] Input: Tap operation

[1061] Output: Task management screen

[1062] Specific action: The user taps the "Task Management" button.

[1063] Step 9:

[1064] The user enters the details of the task to be assigned to a specific robot on the task management screen and taps the "Done" button. The operating device then sends this information to the server.

[1065] Input: Task details

[1066] Output: Task registration information

[1067] Specific actions: The user enters task details and taps the "Done" button.

[1068] Step 10:

[1069] The server stores the received task detail information in a database and transmits the processing results to the operating terminal.

[1070] Input: Task details

[1071] Output: Task registration result

[1072] Specific operation: The server saves the task details in the database and sends the processing results to the operating terminal.

[1073] Step 11:

[1074] The operation terminal updates the task management screen to display the new task.

[1075] Input: Task registration result

[1076] Output: Updated task management screen

[1077] Specific operation: The operation terminal updates the task management screen based on the task registration results and displays the new task.

[1078] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1079] System Configuration

[1080] This invention provides a system that combines a tap-and-release operation that allows users to perform operations intuitively with an emotion engine that recognizes the user's emotions. Specific embodiments of this system and their operation will be described below.

[1081] Basic Configuration

[1082] 1. Operation terminal:

[1083] These are devices that are directly operated by the user, such as smartphones and tablets.

[1084] It has a touch screen that displays the interface.

[1085] It also has a microphone and a camera for capturing the user's voice and facial expressions.

[1086] 2. Server:

[1087] It receives input information and emotion data sent from the operating terminal, processes them, and stores them.

[1088] Access the database and retrieve or register the required information.

[1089] 3. Database:

[1090] It is stored on the server and contains schedule information, task information, and authentication information.

[1091] 4. Interface:

[1092] It is displayed on the operating device and is used by the user with a tap and release operation.

[1093] Events and tasks are visually displayed, allowing for intuitive operation.

[1094] 5. Emotion Engine:

[1095] It analyzes the user's facial expressions, voice, or touch patterns to recognize emotions.

[1096] The interface display and operation methods are dynamically changed based on the recognized emotions.

[1097] Program processing

[1098] The program processing in this system will be explained below.

[1099] Login process

[1100] 1. User: Launch the application, enter your username and password on the login screen, and tap the "Login" button.

[1101] 2. Terminal: Sends an authentication request to the server using the entered username and password.

[1102] 3. Server: Based on the received request, it checks the username and password in the database.

[1103] 4. Server: If authentication is successful, generate an authentication token and send it to the terminal as a response.

[1104] 5. Device: Receives the authentication token and saves it locally. After saving, displays the home screen.

[1105] Schedule adjustment operations

[1106] 1. User: Tap the "Schedule" button on the home screen.

[1107] 2. Device: Display the schedule adjustment screen.

[1108] 3. User: Tap and release the event icon (e.g., meeting icon) on the screen and drag it to the desired time slot (e.g., 9:00-10:00).

[1109] 4. Terminal: Detects the user's tap and release operation and obtains the selected event information and time frame information.

[1110] 5. Terminal: Creates and sends a request to send the acquired information to the server.

[1111] 6. Server: Analyzes the received information and registers it in a database.

[1112] 7. Server: If registration in the database is successful, it sends a success message to the terminal.

[1113] 8. Terminal: Receives the success message and updates the schedule screen to show the new schedule.

[1114] Task Detailing

[1115] 1. User: Tap the "Task Management" button on the schedule adjustment screen.

[1116] 2. Device: Display the task management screen.

[1117] 3. User: Taps the text field to enter new task information.

[1118] 4. User: Enter task details in the text field and tap the "Done" button.

[1119] 5. Terminal: Sends the entered task details to the server.

[1120] 6. Server: Analyzes the received task information and registers it in the database.

[1121] 7. Server: If registration in the database is successful, it sends a success message to the terminal.

[1122] 8. Terminal: Receives the success message and updates the task management screen to show the new task information.

[1123] emotion recognition

[1124] 1. User: Facial expressions and voice are input by the emotion engine during operation.

[1125] 2. Terminal: The emotion engine analyzes the input data and recognizes the user's emotions.

[1126] 3. Terminal: Sends the recognized emotion data to the server.

[1127] 4. Server: Receives emotion data and sends instructions back to the device to dynamically adjust the interface display and operation method.

[1128] 5. Terminal: Dynamically changes the interface based on instructions from the server to adapt to the user's current emotional state.

[1129] Specific examples

[1130] For example, if the emotion engine detects that a user is feeling stressed while scheduling a meeting, that information is sent from the device to the server. The server can then make adjustments, such as simplifying the interface or displaying more relaxing colors and messages. This system significantly reduces the psychological burden on users during operation.

[1131] The above describes an embodiment of the present invention. This system allows users to intuitively utilize AI functions and efficiently manage tasks and schedules through an interface that adapts to their emotional state during operation.

[1132] The processing flow will be explained below.

[1133] Login process

[1134] Step 1:

[1135] User: Launches the application, enters the username and password on the login screen, and taps the "Login" button.

[1136] Step 2:

[1137] Terminal: Sends an authentication request to the server using the entered username and password.

[1138] Step 3:

[1139] Server: Based on the received request, the server checks the username and password in the database.

[1140] Step 4:

[1141] Server: If authentication is successful, generate an authentication token and send it to the terminal as a response.

[1142] Step 5:

[1143] Device: Receives the authentication token and stores it locally. After storing it, displays the home screen.

[1144] Schedule adjustment operations

[1145] Step 1:

[1146] User: Tap the Schedule button on the home screen.

[1147] Step 2:

[1148] Terminal: Display the schedule adjustment screen.

[1149] Step 3:

[1150] User: Tap and release the event icon (e.g., meeting icon) on the screen and drag it to the desired time slot (e.g., 9:00-10:00).

[1151] Step 4:

[1152] Terminal: Detects the user's tap and release operation and obtains the selected event information and time frame information.

[1153] Step 5:

[1154] Terminal: Creates and sends a request to send the acquired information to the server.

[1155] Step 6:

[1156] Server: Analyzes the received information and registers it in a database.

[1157] Step 7:

[1158] Server: If registration in the database is successful, it sends a success message to the terminal.

[1159] Step 8:

[1160] Terminal: Receives the success message and updates the schedule screen to show the new schedule.

[1161] Task Detailing

[1162] Step 1:

[1163] User: Tap the "Task Management" button on the schedule adjustment screen.

[1164] Step 2:

[1165] Terminal: Display the task management screen.

[1166] Step 3:

[1167] User: Taps a text field to enter new task information.

[1168] Step 4:

[1169] User: Enter task details in the text field and tap the "Done" button.

[1170] Step 5:

[1171] Terminal: Sends the entered task details to the server.

[1172] Step 6:

[1173] Server: Analyzes the received task information and registers it in the database.

[1174] Step 7:

[1175] Server: If registration in the database is successful, it sends a success message to the terminal.

[1176] Step 8:

[1177] Terminal: Receives the success message and updates the task management screen to show the new task information.

[1178] emotion recognition

[1179] Step 1:

[1180] User: During operation, emotions are input through the camera and microphone of the operating device.

[1181] Step 2:

[1182] Terminal: The emotion engine analyzes input data (facial expressions, voice, touch patterns) and recognizes the user's emotions.

[1183] Step 3:

[1184] Terminal: Sends the recognized emotion data to the server.

[1185] Step 4:

[1186] Server: Analyzes the received emotional data and generates instructions to dynamically change the interface display and operation method.

[1187] Step 5:

[1188] Server: Sends instructions to the terminal.

[1189] Step 6:

[1190] Terminal: Receives instructions from the server and dynamically changes the interface display and operation method.

[1191] Specific examples

[1192] For example, consider scheduling a meeting. A user launches an application, enters their login information, and logs in. From the home screen, they tap the "Schedule" button to move to the schedule adjustment screen. They drag the meeting icon to the 9:00 to 10:00 time slot, and the device sends this information to the server. The server registers the information in the database and sends a success message to the device. The device updates the schedule adjustment screen to show the meeting in the 9:00 to 10:00 time slot.

[1193] Furthermore, if the emotion engine detects stress from the user's facial expression during operation, it sends this information to the server. The server then generates instructions for relaxing color tones and message displays and sends them to the device. The device then dynamically changes its interface based on these instructions, improving the user's operating experience.

[1194] Example 2

[1195] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1196] Conventional schedule adjustment and task management systems using operating terminals often cause stress to users because they do not take into account the user's emotional state. Furthermore, when operations are complicated, users have difficulty efficiently managing tasks and schedules. Furthermore, there is a need for a method to smoothly access the system while ensuring the security of authentication information.

[1197] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1198] In this invention, the server includes an operation terminal operated by a user, means for processing input information received from the operation terminal, means for accessing a database stored on the server, means for adjusting schedules and detailing tasks by tapping and releasing on the interface, an emotion engine in the operation terminal that analyzes facial expressions and voice to recognize emotions, and means for transmitting the emotion data to the server and dynamically adjusting the display content and operation method of the interface. This enables dynamic adjustment of the interface according to the user's emotional state, providing a user-friendly operation environment and improving the efficiency of schedule adjustment and task management.

[1199] "User" refers to a person who operates the system via an operation terminal.

[1200] An "operation terminal" is a device operated by a user, and refers to devices such as smartphones and tablets.

[1201] "Server" refers to a computer system that processes input information received from an operating terminal and accesses a database.

[1202] "Database" refers to a data storage system that stores schedule information, task information, authentication information, etc.

[1203] "Interface" refers to a user interface that visually displays events and tasks on a screen displayed on an operating terminal.

[1204] "Tap and release operation" refers to an operation in which a user taps and releases their finger on the touchscreen of the operating device.

[1205] An "emotion engine" refers to a system that analyzes facial expressions and voices on an operating device to recognize emotions.

[1206] "Authentication token" refers to data generated by a server that contains authentication information for an authenticated user.

[1207] "Emotion data" refers to data that indicates the user's emotional state as recognized by the emotion engine.

[1208] System Configuration

[1209] This invention provides a system that combines a tap-and-release operation that allows users to perform operations intuitively with an emotion engine that recognizes the user's emotions. The basic configuration of the system is as follows.

[1210] Operation terminal

[1211] An operating terminal is a device that is directly operated by a user, such as a smartphone or tablet. Examples include iPhones and Android devices. These terminals are equipped with a touch screen, microphone, and camera, and display a user interface.

[1212] server

[1213] The server is a computer system that receives, processes, and stores input information and emotion data sent from the operating device. The server accesses a database to retrieve or register the necessary information. As a specific example, Django is used as the server-side framework.

[1214] Database

[1215] The database is a data storage system stored on a server for storing schedule information, task information, and authentication information. Specific examples include SQLite and MySQL.

[1216] Interface

[1217] The interface is displayed on the operating device and is used by the user with tap and release operations, visually indicating events and tasks and allowing intuitive operation.

[1218] Emotion Engine

[1219] An emotion engine is a system that recognizes emotions by analyzing a user's facial expressions, voice, or touch patterns. For example, OpenCV is used for facial recognition, and TensorFlow is used for emotion analysis. The interface display and operation methods are dynamically changed based on the recognized emotions.

[1220] Usage example

[1221] As a specific example, if the emotion engine detects that a user is feeling stressed while scheduling a meeting, that information is sent from the device to the server. The server can then make adjustments, such as simplifying the interface or displaying more relaxing colors and messages. This system significantly reduces the psychological burden on users during operation.

[1222] Example prompts to input to the generative AI model:

[1223] "How can I recognize a user's emotions while using a scheduling app and adjust the interface when the user feels stressed?"

[1224] The above description concretely illustrates the embodiments of the present invention. This system allows users to intuitively utilize AI functions and efficiently manage tasks and schedules through an interface that adapts to the user's emotional state during operation.

[1225] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1226] Login process

[1227] Step 1:

[1228] User: Launches the application, enters the username and password on the login screen, and taps the "Login" button.

[1229] Input: Username and Password

[1230] Output: Login operation trigger

[1231] Step 2:

[1232] Terminal: Sends an authentication request to the server using the entered username and password.

[1233] Specifically, the terminal generates an HTTP POST request and sends it to the server's authentication API endpoint.

[1234] Input: Username and Password

[1235] Output: Authentication request

[1236] Step 3:

[1237] Server: Based on the received request, the server checks the username and password in the database.

[1238] Specifically, the server runs a database query to verify that the username and hashed password match.

[1239] Input: Authentication Request

[1240] Output: Authentication result

[1241] Step 4:

[1242] Server: If authentication is successful, generate an authentication token and send it to the terminal as a response.

[1243] Specifically, it generates a JWT token along with a success message and returns it as an HTTP response.

[1244] Input: Authentication result

[1245] Output: Authentication token

[1246] Step 5:

[1247] Device: Receives the authentication token and stores it locally. After storing it, displays the home screen.

[1248] Specifically, the token is saved in local storage and the screen transitions to the next screen.

[1249] Input: Authentication Token

[1250] Output: Home screen display

[1251] Schedule adjustment operations

[1252] Step 1:

[1253] User: Tap the Schedule button on the home screen.

[1254] Input: User taps

[1255] Output: Display of schedule screen

[1256] Step 2:

[1257] Terminal: Display the schedule adjustment screen.

[1258] Input: Schedule screen display request

[1259] Output: Schedule adjustment screen

[1260] Step 3:

[1261] User: Tap and release the event icon (e.g., meeting icon) on the screen and drag it to the desired time slot (e.g., 9:00-10:00).

[1262] Input: Tap and release

[1263] Output: Drag operation completed

[1264] Step 4:

[1265] Terminal: Detects the user's tap and release operation and obtains the selected event information and time frame information.

[1266] Specifically, the event ID and time frame are stored as an internal data structure.

[1267] Input: Drag operation complete

[1268] Output: Event information and time slot information

[1269] Step 5:

[1270] Terminal: Creates and sends a request to send the acquired information to the server.

[1271] Specifically, it generates an HTTP POST request and sends it to the server's schedule API endpoint.

[1272] Input: Event information and time slot information

[1273] Output: Reschedule request

[1274] Step 6:

[1275] Server: Analyzes the received information and registers it in a database.

[1276] The specific operation is to execute an INSERT or UPDATE query on the database.

[1277] Input: Schedule change request

[1278] Output: Database update results

[1279] Step 7:

[1280] Server: If registration in the database is successful, it sends a success message to the terminal.

[1281] Specifically, a success status is returned in the HTTP response.

[1282] Input: Database update results

[1283] Output: Success message

[1284] Step 8:

[1285] Terminal: Receives the success message and updates the schedule screen to show the new schedule.

[1286] Specific operations include updating local data and redrawing the user interface.

[1287] Input: Success message

[1288] Output: Updated schedule contents

[1289] Task Detailing

[1290] Step 1:

[1291] User: Tap the "Task Management" button on the schedule adjustment screen.

[1292] Input: User taps

[1293] Output: Display of task management screen

[1294] Step 2:

[1295] Terminal: Display the task management screen.

[1296] Input: Request to display the task management screen

[1297] Output: Task management screen

[1298] Step 3:

[1299] User: Taps a text field to enter new task information.

[1300] Input: Tap operation

[1301] Output: Focus on text field

[1302] Step 4:

[1303] User: Enter task details in the text field and tap the "Done" button.

[1304] Input: Task information

[1305] Output: Task information input completed

[1306] Step 5:

[1307] Terminal: Sends the entered task details to the server.

[1308] Specifically, it generates an HTTP POST request and sends it to the server's task management API endpoint.

[1309] Input: Completed entering task information

[1310] Output: Task add request

[1311] Step 6:

[1312] Server: Analyzes the received task information and registers it in the database.

[1313] As a specific operation, an INSERT query is executed on the database.

[1314] Input: Task Add Request

[1315] Output: Database update results

[1316] Step 7:

[1317] Server: If registration in the database is successful, it sends a success message to the terminal.

[1318] Specifically, a success status is returned in the HTTP response.

[1319] Input: Database update results

[1320] Output: Success message

[1321] Step 8:

[1322] Terminal: Receives the success message and updates the task management screen to show the new task information.

[1323] Specific operations include updating local data and redrawing the user interface.

[1324] Input: Success message

[1325] Output: Updated task information

[1326] emotion recognition

[1327] Step 1:

[1328] User: Facial expressions and voice are input by the emotion engine during operation.

[1329] Input: Facial expressions and voice data

[1330] Output: Input to the emotion engine

[1331] Step 2:

[1332] Terminal: The emotion engine analyzes the input data and recognizes the user's emotions.

[1333] Specifically, it uses machine learning models to classify emotions.

[1334] Input: Facial expressions and voice data

[1335] Output: Recognized emotion data

[1336] Step 3:

[1337] Terminal: Sends the recognized emotion data to the server.

[1338] Specifically, it creates emotion data in JSON format and generates and sends an HTTP POST request.

[1339] Input: Recognized emotion data

[1340] Output: Sending emotion data

[1341] Step 4:

[1342] Server: Receives emotion data and sends instructions back to the device to dynamically adjust the interface display and operation method.

[1343] Specifically, it generates UI change instructions according to the emotion and returns them as an HTTP response.

[1344] Input: Emotion data

[1345] Output: UI change instructions

[1346] Step 5:

[1347] Terminal: Receives instructions from the server and dynamically changes the interface to adapt to the user's current emotional state.

[1348] Specific actions include changing the layout and color of the interface.

[1349] Input: UI change instructions

[1350] Output: Dynamically changed interface

[1351] (Application example 2)

[1352] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1353] Conventional systems lack an operational interface that takes user emotions into account, which means that operability declines when users feel stressed. Furthermore, there is a lack of systems to improve individual customer experiences in brick-and-mortar stores, and there is also the problem that customization suggestions based on user emotions have not yet been fully implemented.

[1354] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1355] In this invention, the server includes an operation terminal operated by a user, a central processing unit that processes input information received from the operation terminal, an information storage device that stores information stored in the central processing unit, a display unit that displays tasks on the operation terminal, a unit that adjusts schedules and subdivides tasks through selection operations on the display unit, an emotion recognition unit that analyzes the user's facial expression and voice data, and a unit that dynamically changes the content of the display unit based on the emotion identified by the emotion recognition unit. This makes it possible to dynamically adjust the interface according to the user's emotion and improve the individual customer experience in physical stores.

[1356] An "operation terminal" is a device that is directly operated by a user, such as a smartphone or tablet.

[1357] The "central processing unit" is a device that processes input information received from an operation terminal, and is usually a computer system called a server.

[1358] "Information storage device" means a device for storing data saved in a central processing unit, and includes a database.

[1359] The "display means" is an interface for displaying tasks and events on the operating terminal, and is generally a touch screen display.

[1360] The "selection operation" refers to an action in which a user selects a task or event on the display means, and includes finger movements such as tapping and swiping.

[1361] "Emotion recognition means" is a technology that identifies emotions by analyzing a user's facial expressions and voice data, and machine learning models are often used.

[1362] The "dynamically changing means" refers to a function of adjusting the content of the display means in real time based on the emotion identified by the emotion recognition means.

[1363] The embodiment of the present invention is a system comprising an operation terminal operated by a user, a central processing unit, an information storage device, a display means, a selection operation means, an emotion recognition means, and a dynamic change means. A specific embodiment of this system will be described in detail below.

[1364] System Configuration

[1365] 1. Operation terminal: A device that is directly operated by the user, such as a smartphone or tablet. It is equipped with a touch screen, camera, and microphone, and can capture the user's facial expressions and voice.

[1366] 2. Central Processing Unit: A computer system that processes input information received from the operating terminal, usually called a server. It integrates multiple input information and retrieves and registers data from the information storage device.

[1367] 3. Information storage device: A device that includes a database and stores schedule information, task information, user authentication information, emotion data, etc.

[1368] 4. Display: An interface displayed on the operating device, providing a visual display to intuitively show tasks and events. It accepts operations such as tapping and swiping.

[1369] 5. Selection operation means: This function detects tap and release operations performed by the user on the display means and allows for schedule adjustments and task subdivision.

[1370] 6. Emotion recognition: Analyzes facial expressions and voice data acquired from the smartphone camera and microphone to identify the user's emotions. Uses machine learning models (e.g., TensorFlow, PyTorch).

[1371] 7. Dynamic modification: This function adjusts the content of the display means in real time based on the emotional data identified by the emotion recognition means. The interface is modified in a way that adapts to the user's current emotional state.

[1372] Program processing explanation

[1373] 1. User: Uses a smartphone to launch the application and operate the interface shown on the display means.

[1374] 2. Device: Detects selection operations such as taps and swipes performed by the user. Also, captures the user's facial expressions and voice using a camera and microphone. This data is sent to the emotion recognition system.

[1375] 3. Emotion recognition means: Analyzes facial expressions and voice data acquired using a machine learning model to identify the user's emotions. The identified emotion data is sent to the central processing unit.

[1376] 4. Central processing unit: After receiving the emotion data, it retrieves related data from the information storage device and generates content to be sent to the display means. It also generates product suggestions and schedule information according to the emotion.

[1377] 5. Display means: Based on the data sent from the central processing unit, the interface content is dynamically changed and displayed to the user, such as special product offers, event information, or relaxing color themes.

[1378] Specific examples

[1379] For example, if a user stops in front of a particular product shelf in a physical store and looks confused, the smartphone's camera and microphone will capture this. The emotion recognition means will identify the emotion as "confusion" and send that data to the central processing unit. The central processing unit will then generate an interface that provides the confused user with detailed product information and usage instructions, and display this on the display means in real time.

[1380] Prompt Sentence Examples

[1381] "Users' emotions are confusing. Can you recommend any products?"

[1382] This system allows the interface to dynamically adjust based on the user's emotions, significantly improving the personalized customer experience in physical stores.

[1383] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1384] Step 1:

[1385] The user starts the application using a smartphone, enters the username and password on the login screen, and taps the "Login" button.

[1386] Input: Username, Password

[1387] Output: Generated authentication request

[1388] Specifically, it detects input on the smartphone's touchscreen and creates an authentication request.

[1389] Step 2:

[1390] The device sends an authentication request to the server.

[1391] Input: Authentication request

[1392] Output: Send data to the server

[1393] Specifically, the network function of the smartphone is used to send authentication request data to the server.

[1394] Step 3:

[1395] Based on the request received by the server, the information storage device checks the user name and password.

[1396] Input: Authentication request (username, password)

[1397] Output: Authentication result (success or failure)

[1398] Specifically, it performs a database query and compares the received data with the stored data.

[1399] Step 4:

[1400] If the server is successful in the authentication, it generates an authentication token and sends it to the terminal as a response.

[1401] Input: Authentication result (success)

[1402] Output: Authentication token

[1403] Specifically, the authentication token is generated using a random number generation algorithm and sent to the terminal.

[1404] Step 5:

[1405] The device receives the authentication token, stores it locally, and displays the home screen.

[1406] Input: Authentication token

[1407] Output: Home screen display

[1408] Specifically, it stores the token in local storage and updates the application's UI component to display the home screen.

[1409] Step 6:

[1410] As the user walks around the store, the device uses a camera and microphone to capture the user's facial expressions and voice.

[1411] Input: Camera video, audio data

[1412] Output: Captured data

[1413] Specifically, it periodically collects data from the camera and microphone sensors.

[1414] Step 7:

[1415] An emotion recognition means in the terminal analyzes the captured data and identifies the user's emotion.

[1416] Input: Captured data (facial expression, voice)

[1417] Output: Recognized emotion data

[1418] Specifically, it uses machine learning models to analyze data and identify specific emotions.

[1419] Step 8:

[1420] The device transmits the recognized emotion data to the server.

[1421] Input: Emotion data

[1422] Output: Send emotion data to the server

[1423] As a specific operation, emotion data is transmitted via a network.

[1424] Step 9:

[1425] The server receives the emotion data and uses a generative AI model to generate data for appropriate product suggestions and interface changes.

[1426] Input: Emotion data

[1427] Output: Proposal data, interface change data

[1428] Specifically, the prompt sentence "The user is confused. Please recommend some products." is input into the AI ​​model, and the generated suggestions are obtained.

[1429] Step 10:

[1430] The server transmits the generated proposal data to the terminal.

[1431] Input: Proposal data

[1432] Output: Sending data to the terminal

[1433] As a specific operation, the proposal data is transmitted to the terminal via the network.

[1434] Step 11:

[1435] The terminal receives the proposal data and dynamically adjusts and presents an interface to the user using the display means.

[1436] Input: Proposal data

[1437] Output: Dynamically adjusted interface

[1438] Specifically, it updates interface elements such as display content, color themes, and messages and displays them to the user.

[1439] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1440] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1441] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1442] [Third embodiment]

[1443] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1444] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1445] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1446] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1447] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1448] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1449] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1450] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1451] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1452] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1453] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1454] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1455] The purpose of this invention is to utilize an interface that users can operate intuitively to more effectively and easily utilize the functions of artificial intelligence (AI). A specific embodiment of this system and its operation will be described below.

[1456] System Configuration

[1457] The system mainly consists of the following components:

[1458] 1. Operation terminal

[1459] A device that is directly operated by the user. Examples include smartphones and tablet devices.

[1460] It has a touch screen that displays the interface.

[1461] 2. Server

[1462] It receives input information sent from the operating terminal, processes it, and stores it.

[1463] Access the database and retrieve or register the required information.

[1464] 3. Database

[1465] It is stored on the server and stores schedule information, task information, etc.

[1466] 4. Interface

[1467] It is displayed on the operating device and is used by the user with a tap and release operation.

[1468] Events and tasks are visually displayed, allowing for intuitive operation.

[1469] Program processing

[1470] The following describes the program processing in this system. The processing of each function is described with the subject being either the operating terminal, the server, or the user.

[1471] Login process

[1472] 1. User: Launches the application and enters username and password on the login screen.

[1473] 2. Terminal: Sends the entered username and password to the server.

[1474] 3. Server: Validates the username and password against the database and, if authentication is successful, generates an authentication token.

[1475] 4. Server: Sends the authentication token to the operating terminal.

[1476] 5. Device: Stores the received authentication token locally and displays the user's home screen.

[1477] Schedule adjustment operations

[1478] 1. User: Tap the "Schedule" button from the home screen.

[1479] 2. Terminal: Displays the schedule adjustment screen and allows the user to select an event.

[1480] 3. User: Tap and release the event icon on the interface and drag it to the desired time frame.

[1481] 4. Terminal: Obtains the information of the selected event and time frame and sends it to the server.

[1482] 5. Server: Registers the received information in a database and sends the processing results to the terminal.

[1483] 6. Terminal: Receives the processing results, updates the schedule screen, and displays the registered details.

[1484] Task Detailing

[1485] 1. User: Tap the "Task Management" button on the schedule adjustment screen.

[1486] 2. Terminal: Displays the task management screen and provides text fields for entering details.

[1487] 3. User: Enter task details in the text field and tap the "Done" button.

[1488] 4. Terminal: Sends the entered task information to the server.

[1489] 5. Server: Registers the received information in a database and sends the processing results to the terminal.

[1490] 6. Terminal: Receives the processing results and updates the task management screen to display the new task.

[1491] Specific examples

[1492] For example, consider a user who wants to schedule a meeting from 9:00 to 10:00. The user first launches the application and enters their login information. After logging in, they navigate to the home screen and tap the "Schedule" button. The schedule adjustment screen appears, and the user drags the meeting icon to the 9:00 to 10:00 time slot. This operation causes the device to send the event information and time slot information to the server, which then registers it in the database. As a result, the end user can see that the schedule has been updated appropriately.

[1493] The above describes an embodiment of the present invention. This system allows users to intuitively utilize AI functions and efficiently adjust schedules and refine tasks.

[1494] The processing flow will be explained below.

[1495] Login process

[1496] Step 1:

[1497] User: Launches the application, enters the username and password on the login screen, and taps the "Login" button.

[1498] Step 2:

[1499] Terminal: Sends an authentication request to the server using the entered username and password.

[1500] Step 3:

[1501] Server: Based on the received request, the server checks the username and password in the database.

[1502] Step 4:

[1503] Server: If authentication is successful, generate an authentication token and send it to the terminal as a response.

[1504] Step 5:

[1505] Device: Receives the authentication token and stores it locally. After storing it, displays the home screen.

[1506] Schedule adjustment operations

[1507] Step 1:

[1508] User: Tap the Schedule button on the home screen.

[1509] Step 2:

[1510] Terminal: Display the schedule adjustment screen.

[1511] Step 3:

[1512] User: Tap and release the event icon (e.g., meeting icon) on the screen and drag it to the desired time slot (e.g., 9:00-10:00).

[1513] Step 4:

[1514] Terminal: Detects the user's tap and release operation and obtains the selected event information and time frame information.

[1515] Step 5:

[1516] Terminal: Creates and sends a request to send the acquired information to the server.

[1517] Step 6:

[1518] Server: Analyzes the received information and registers it in a database.

[1519] Step 7:

[1520] Server: If registration in the database is successful, it sends a success message to the terminal.

[1521] Step 8:

[1522] Terminal: Receives the success message and updates the schedule screen to show the new schedule.

[1523] Task Detailing

[1524] Step 1:

[1525] User: Tap the "Task Management" button on the schedule adjustment screen.

[1526] Step 2:

[1527] Terminal: Display the task management screen.

[1528] Step 3:

[1529] User: Taps a text field to enter new task information.

[1530] Step 4:

[1531] User: Enter task details in the text field and tap the "Done" button.

[1532] Step 5:

[1533] Terminal: Sends the entered task details to the server.

[1534] Step 6:

[1535] Server: Analyzes the received task information and registers it in the database.

[1536] Step 7:

[1537] Server: If registration in the database is successful, it sends a success message to the terminal.

[1538] Step 8:

[1539] Terminal: Receives the success message and updates the task management screen to show the new task information.

[1540] Example 1

[1541] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1542] In conventional systems, it was difficult for users to efficiently adjust schedules and detail tasks using an intuitive interface. The authentication process was also cumbersome, resulting in a poor user experience. Furthermore, there was a lack of a mechanism for reliably and securely transmitting task and event information to the server.

[1543] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1544] In this invention, the server includes means for processing input information received from the operation terminal, a database stored in the operation terminal, an interface means displayed on the operation terminal for displaying events and tasks, means for adjusting schedules and refining tasks through tap-and-release operations on the interface, means for verifying authentication information received from the operation terminal, generating an authentication token and transmitting it to the operation terminal, and means for transmitting event information and time slot information acquired based on the tap-and-release operations to the server. This allows users to effectively utilize AI functions through intuitive operations, enabling efficient schedule adjustments and task refinement.

[1545] An "information processing terminal" is a device that is operated by a user to input and display information.

[1546] The "arithmetic unit" is a device that processes input information received from an information processing terminal, and performs database management and various other operations.

[1547] An "information storage device" is a storage medium connected to a computing device for storing and managing data.

[1548] The "display means" is an interface that is built into the information processing terminal and that visually shows the work contents and schedule to the user.

[1549] The term "instruction operation" refers to an operation performed with a finger, a stylus, or the like using the display means of the information processing terminal.

[1550] "Authentication information" refers to information such as a user name and password that a user enters when using an information processing terminal.

[1551] An "authentication token" is data that indicates the authentication of a user and is generated when authentication information is successfully verified.

[1552] "Work information" is detailed information about tasks and events.

[1553] "Time information" is information about the time period in which a task or event is executed.

[1554] This invention relates to a system for effectively and easily adjusting schedules and detailing tasks via an intuitive interface on an information processing terminal (e.g., a smartphone or tablet) operated by a user.

[1555] System Configuration

[1556] The system mainly consists of the following components:

[1557] 1. Information processing terminal

[1558] It is a device that is directly operated by the user.

[1559] Examples include smartphones and tablet devices, which have touchscreens.

[1560] This terminal is equipped with a display means for visually showing the work content and schedule.

[1561] 2. Arithmetic device

[1562] It processes input information received from the information processing terminal and performs database management and various operations.

[1563] This applies to web servers or cloud servers.

[1564] 3. Information storage device

[1565] It is a storage medium connected to a computing device for storing and managing data.

[1566] A database management system (RDBMS) is used.

[1567] System Functions and Operation

[1568] Examples of the main functions of this system include login processing, schedule adjustment, and task detailing. The operation of each function is as follows:

[1569] Login process

[1570] 1. The user launches the application and enters authentication information (username and password) on the login screen.

[1571] 2. The information processing terminal transmits the input authentication information to the arithmetic unit.

[1572] 3. The computing device checks the authentication information against information stored in a database and, if successful, generates an authentication token.

[1573] 4. The computing device transmits the authentication token to the information processing terminal.

[1574] 5. The information processing terminal stores the authentication token in local storage and displays the user's home screen.

[1575] Schedule adjustment

[1576] 1. The user taps the "Schedule" button on the home screen.

[1577] 2. The information processing terminal displays the schedule screen, and the user drags and drops the schedule into the desired time slot.

[1578] 3. The information processing terminal transmits information about the selected schedule and time slot to the computing device.

[1579] 4. The arithmetic unit registers the received information in a database and returns the processing results to the information processing terminal.

[1580] 5. The information processing terminal receives the processing results and updates the schedule screen.

[1581] Task Detailing

[1582] 1. The user taps the "Task Management" button on the schedule adjustment screen.

[1583] 2. The information processing terminal displays the task management screen, and the user enters the task details.

[1584] 3. The information processing terminal transmits the input task information to the arithmetic unit.

[1585] 4. The arithmetic unit registers the received information in a database and returns the processing results to the information processing terminal.

[1586] 5. The information processing terminal receives the processing results and updates the task management screen.

[1587] Specific examples

[1588] For example, if a user "reschedules a meeting from 9 to 10":

[1589] 1. The user launches the application and enters their username and password at the login screen.

[1590] 2. The user taps the Schedule button on the home screen.

[1591] 3. The information processing terminal displays the schedule screen.

[1592] 4. The user drags and drops the meeting icon into the 9:00 to 10:00 time slot.

[1593] 5. The information processing terminal transmits the information about the selected meeting and the new time slot information to the computing device.

[1594] 6. The arithmetic unit registers the received data in a database and returns the processing results to the information processing terminal.

[1595] 7. The information processing terminal receives the processing results and displays that the schedule has been appropriately updated.

[1596] Example prompts for generative AI models

[1597] "Generate a sentence that explains the steps a user takes to reschedule a meeting from 9:00 to 10:00. The steps include launching the application, logging in, displaying the schedule screen, selecting the meeting icon, moving to the time slot, sending the data to the server, registering it in the database, and checking for updates."

[1598] This system allows users to intuitively utilize AI functions to efficiently adjust schedules and detail tasks.

[1599] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1600] Login process

[1601] Step 1:

[1602] A user launches an application and is directed to a login screen, where they enter their authentication information (username and password).

[1603] Input: Username and Password

[1604] Output: Create login request

[1605] Step 2:

[1606] The device sends the entered username and password to the server using the HTTPS protocol, which prevents unauthorized interception of the data.

[1607] Input: Username and Password

[1608] Output: HTTPS request

[1609] Step 3:

[1610] The server checks the received username and password against the information stored in the database, using an SQL query to look up the user information and confirm a match.

[1611] Input: Username and password in the HTTPS request

[1612] Output: Authentication result (success or failure)

[1613] Step 4:

[1614] If the server is successful in authenticating, it generates an authentication token, such as a JWT (JSON Web Token), which is used to ensure that subsequent requests are valid.

[1615] Input: Authentication result (success)

[1616] Output: Authentication token

[1617] Step 5:

[1618] The server sends the generated authentication token to the terminal using the HTTPS protocol.

[1619] Input: Authentication Token

[1620] Output: HTTPS response

[1621] Step 6:

[1622] The device stores the received authentication token in local storage and prepares to display the user's home screen.

[1623] Input: HTTPS response containing an authentication token

[1624] Output: Home screen display

[1625] Schedule adjustment operations

[1626] Step 1:

[1627] The user taps the "Schedule" button from the home screen, which takes the user to the schedule adjustment screen.

[1628] Input: Home screen interaction

[1629] Output: Schedule screen request

[1630] Step 2:

[1631] The device displays a schedule adjustment screen, providing a calendar and time slot interface.

[1632] Input: Schedule screen request

[1633] Output: Schedule adjustment screen

[1634] Step 3:

[1635] Users can visually change the schedule by dragging and dropping event icons onto the desired time slots on the interface.

[1636] Input: Drag and drop of event icons

[1637] Output: New timeframe selection information

[1638] Step 4:

[1639] The device sends the selected event information and new time slot information to the server in JSON format, which is used for subsequent processing.

[1640] Input: New timeframe selection information

[1641] Output: JSON format data

[1642] Step 5:

[1643] The server parses the received JSON data and executes SQL queries to register the relationship between events and time frames in the database.

[1644] Input: JSON format data

[1645] Output: Registration results in the database

[1646] Step 6:

[1647] The server returns the processing result (success or failure) to the terminal along with a status code.

[1648] Input: Registration results to the database

[1649] Output: Response with status code

[1650] Step 7:

[1651] The device receives the status code and updates the schedule screen accordingly. If successful, the new schedule information is displayed.

[1652] Input: Response with status code

[1653] Output: Updated schedule screen

[1654] Task Detailing

[1655] Step 1:

[1656] The user taps the "Task Management" button on the schedule adjustment screen, which takes the user to the task management screen.

[1657] Input: Schedule adjustment screen interaction

[1658] Output: Task management screen request

[1659] Step 2:

[1660] The device displays a task management screen, providing text fields and options for the user to enter task details.

[1661] Input: Task management screen request

[1662] Output: Task management screen

[1663] Step 3:

[1664] The user fills in the task details in the input fields and taps the "Done" button, which creates a new task.

[1665] Input: Enter task details

[1666] Output: New task information

[1667] Step 4:

[1668] The device sends new task information in JSON format to the server.

[1669] Input: New task information

[1670] Output: JSON format data

[1671] Step 5:

[1672] The server parses the received JSON data and executes an SQL query to register new task information in the database.

[1673] Input: JSON format data

[1674] Output: Registration results in the database

[1675] Step 6:

[1676] The server returns the processing result (success or failure) to the terminal along with a status code.

[1677] Input: Registration results to the database

[1678] Output: Response with status code

[1679] Step 7:

[1680] The device receives the status code and updates the task management screen accordingly, displaying the new task information if successful.

[1681] Input: Response with status code

[1682] Output: Updated task management screen

[1683] (Application example 1)

[1684] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1685] Managing robot operation schedules and task allocation within a factory is complex and time-consuming, placing a significant burden on managers. This burden increases even more when real-time adjustments and changes are required. Current systems lack an intuitive interface for performing these operations, making efficient management difficult. Therefore, there is a need for a method for efficiently managing robot operation schedules and task allocation within a factory using an intuitive interface.

[1686] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1687] In this invention, the server includes an operation terminal operated by a user, means for processing input information received from the operation terminal, a database stored in the server, an interface that indicates events and tasks and is displayed on the operation terminal, means for adjusting schedules and refining tasks by tapping and releasing on the interface, and means for intuitively managing robot operation schedules and task allocation within a factory via the operation terminal. This enables a user to efficiently manage robot operation schedules and task allocation within a factory using an interface that can be intuitively operated.

[1688] A "user" is a person who operates the system using an operation terminal.

[1689] An "operation terminal" is a device used by a user to adjust schedules and manage tasks, and includes smartphones, tablet terminals, and the like.

[1690] A "server" is a device that processes input information sent from an operating terminal and manages data stored in a database.

[1691] A "database" is a system that is stored on a server and stores data such as schedule information and task information.

[1692] An "interface" is a screen that is displayed on an operation terminal and allows a user to visually check events and tasks and perform operations.

[1693] A "tap and release operation" is a series of operations in which a user taps, drags, and then releases their finger on the touch screen of the operating terminal.

[1694] "Schedule adjustment" refers to an operation in which a user sets or changes the time frame of an event or task on the interface.

[1695] "Task elaboration" is an operation in which a user inputs and manages detailed information about a specific task.

[1696] A "robot" is an automated mechanical device used in factories that has the ability to perform various tasks.

[1697] A "robot operation schedule" is a plan that indicates what operations a robot will perform at what time periods.

[1698] "Task allocation" means assigning specific tasks to each robot.

[1699] This invention provides a system that allows intuitive management of robot operation schedules and task allocation within a factory. The main elements of the system are an operation terminal, a server, a database, and an interface. Specific mechanisms include login processing, schedule adjustment, and task detailing.

[1700] System Configuration

[1701] The system consists of the following elements:

[1702] 1. Operation terminal:

[1703] A device operated by a user, such as a smartphone or tablet.

[1704] It is equipped with a touch screen that allows users to operate it intuitively.

[1705] 2. Server:

[1706] This is a device that receives, processes, and saves input information sent from an operating terminal.

[1707] Access a database to manage schedule and task-related information.

[1708] 3. Database:

[1709] It is a system that stores schedule information, task information, etc. on a server.

[1710] 4. Interface:

[1711] This screen is displayed on the operation terminal and allows the user to visually check and operate events and tasks.

[1712] You can adjust schedules and detail tasks with a tap and release operation.

[1713] Program processing

[1714] Hardware and Software

[1715] Hardware:

[1716] Smartphone: A device running an operating system such as iOS or Android.

[1717] Server: A device that processes data and manages databases.

[1718] software:

[1719] Flask: A Python web framework for running server-side programs.

[1720] SQLite: A lightweight relational database management system.

[1721] Data processing and calculation

[1722] The server processes the authentication information received from the operation terminal and generates an authentication token. It also processes the event information and time slot information received from the operation terminal and registers it in a database. Specifically, when a user adjusts a schedule or details a task on the operation terminal, the operation terminal sends this data to the server. The server processes the received data, stores it in the database, and returns the results to the operation terminal. This allows users to intuitively manage the operation schedules and task allocation of robots in the factory.

[1723] Specific examples

[1724] For example, if a factory manager sets the operation schedule for Robot A from 1:00 to 2:00 and adds the task "inspection work," the process is as follows:

[1725] 1. Launch the app and log in by entering your username and password.

[1726] 2. Tap the "Schedule" button on the home screen and drag the Robot A icon to the 1:00 to 2:00 time slot.

[1727] 3. Next, tap the "Task Management" button, enter "Inspection Work," and tap "Done."

[1728] Prompt Sentence Examples

[1729] An example prompt is:

[1730] "Please generate a program to schedule Robot A from 1:00 to 2:00 and add 'Inspection Work' to the task."

[1731] This allows users to easily manage schedules and tasks for robots in their factories, improving factory operational efficiency.

[1732] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1733] Step 1:

[1734] The user starts up the operation terminal and enters the user name and password on the login screen. This input information is sent from the operation terminal to the server.

[1735] Input: Username, Password

[1736] Output: Authentication token, login success / failure message

[1737] Specific actions: The user uses the touchscreen to enter login information and taps the submit button.

[1738] Step 2:

[1739] The server verifies the received username and password in the database. If authentication is successful, it generates an authentication token and sends it to the operating terminal.

[1740] Input: Username, Password

[1741] Output: Authentication token

[1742] Specific operation: The server queries the database to verify the user information. If authentication is successful, the server generates an authentication token and sends it to the operating terminal.

[1743] Step 3:

[1744] The operation terminal locally stores the received authentication token and displays the user's home screen.

[1745] Input: Authentication Token

[1746] Output: Home screen

[1747] Specific operation: The operating terminal stores the authentication token and displays the home screen.

[1748] Step 4:

[1749] The user taps the "Schedule" button on the home screen to display the schedule adjustment screen.

[1750] Input: Tap operation

[1751] Output: Schedule adjustment screen

[1752] Specific actions: The user taps the "Schedule" icon on the home screen.

[1753] Step 5:

[1754] The user drags and drops a specific robot icon into the desired time slot on the schedule adjustment screen. The operating terminal acquires event information and time slot information based on this operation.

[1755] Input: Tap and release, drag

[1756] Output: Event information, time slot information

[1757] Specific behavior: The user touches the robot icon, drags it to the desired time frame, and releases it.

[1758] Step 6:

[1759] The operation terminal sends the acquired event information and time slot information to the server, which stores the received information in a database and updates the schedule based on that information.

[1760] Input: Event information, time slot information

[1761] Output: Schedule update information

[1762] Specific operation: The operation terminal sends the event information and time frame information to the server, and the server executes an update query on the database.

[1763] Step 7:

[1764] The server transmits the processing result to the operation terminal, and the operation terminal updates the schedule adjustment screen to display the new schedule.

[1765] Input: Schedule Update Information

[1766] Output: Updated schedule screen

[1767] Specific operation: The server sends schedule update information to the operation terminal, and the operation terminal updates the schedule screen.

[1768] Step 8:

[1769] The user taps the "Task Management" button on the schedule adjustment screen to display the task management screen.

[1770] Input: Tap operation

[1771] Output: Task management screen

[1772] Specific action: The user taps the "Task Management" button.

[1773] Step 9:

[1774] The user enters the details of the task to be assigned to a specific robot on the task management screen and taps the "Done" button. The operating device then sends this information to the server.

[1775] Input: Task details

[1776] Output: Task registration information

[1777] Specific actions: The user enters task details and taps the "Done" button.

[1778] Step 10:

[1779] The server stores the received task detail information in a database and transmits the processing results to the operating terminal.

[1780] Input: Task details

[1781] Output: Task registration result

[1782] Specific operation: The server saves the task details in the database and sends the processing results to the operating terminal.

[1783] Step 11:

[1784] The operation terminal updates the task management screen to display the new task.

[1785] Input: Task registration result

[1786] Output: Updated task management screen

[1787] Specific operation: The operation terminal updates the task management screen based on the task registration results and displays the new task.

[1788] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1789] System Configuration

[1790] This invention provides a system that combines a tap-and-release operation that allows users to perform operations intuitively with an emotion engine that recognizes the user's emotions. Specific embodiments of this system and their operation will be described below.

[1791] Basic Configuration

[1792] 1. Operation terminal:

[1793] These are devices that are directly operated by the user, such as smartphones and tablets.

[1794] It has a touch screen that displays the interface.

[1795] It also has a microphone and a camera for capturing the user's voice and facial expressions.

[1796] 2. Server:

[1797] It receives input information and emotion data sent from the operating terminal, processes them, and stores them.

[1798] Access the database and retrieve or register the required information.

[1799] 3. Database:

[1800] It is stored on the server and contains schedule information, task information, and authentication information.

[1801] 4. Interface:

[1802] It is displayed on the operating device and is used by the user with a tap and release operation.

[1803] Events and tasks are visually displayed, allowing for intuitive operation.

[1804] 5. Emotion Engine:

[1805] It analyzes the user's facial expressions, voice, or touch patterns to recognize emotions.

[1806] The interface display and operation methods are dynamically changed based on the recognized emotions.

[1807] Program processing

[1808] The program processing in this system will be explained below.

[1809] Login process

[1810] 1. User: Launch the application, enter your username and password on the login screen, and tap the "Login" button.

[1811] 2. Terminal: Sends an authentication request to the server using the entered username and password.

[1812] 3. Server: Based on the received request, it checks the username and password in the database.

[1813] 4. Server: If authentication is successful, generate an authentication token and send it to the terminal as a response.

[1814] 5. Device: Receives the authentication token and saves it locally. After saving, displays the home screen.

[1815] Schedule adjustment operations

[1816] 1. User: Tap the "Schedule" button on the home screen.

[1817] 2. Device: Display the schedule adjustment screen.

[1818] 3. User: Tap and release the event icon (e.g., meeting icon) on the screen and drag it to the desired time slot (e.g., 9:00-10:00).

[1819] 4. Terminal: Detects the user's tap and release operation and obtains the selected event information and time frame information.

[1820] 5. Terminal: Creates and sends a request to send the acquired information to the server.

[1821] 6. Server: Analyzes the received information and registers it in a database.

[1822] 7. Server: If registration in the database is successful, it sends a success message to the terminal.

[1823] 8. Terminal: Receives the success message and updates the schedule screen to show the new schedule.

[1824] Task Detailing

[1825] 1. User: Tap the "Task Management" button on the schedule adjustment screen.

[1826] 2. Device: Display the task management screen.

[1827] 3. User: Taps the text field to enter new task information.

[1828] 4. User: Enter task details in the text field and tap the "Done" button.

[1829] 5. Terminal: Sends the entered task details to the server.

[1830] 6. Server: Analyzes the received task information and registers it in the database.

[1831] 7. Server: If registration in the database is successful, it sends a success message to the terminal.

[1832] 8. Terminal: Receives the success message and updates the task management screen to show the new task information.

[1833] emotion recognition

[1834] 1. User: Facial expressions and voice are input by the emotion engine during operation.

[1835] 2. Terminal: The emotion engine analyzes the input data and recognizes the user's emotions.

[1836] 3. Terminal: Sends the recognized emotion data to the server.

[1837] 4. Server: Receives emotion data and sends instructions back to the device to dynamically adjust the interface display and operation method.

[1838] 5. Terminal: Dynamically changes the interface based on instructions from the server to adapt to the user's current emotional state.

[1839] Specific examples

[1840] For example, if the emotion engine detects that a user is feeling stressed while scheduling a meeting, that information is sent from the device to the server. The server can then make adjustments, such as simplifying the interface or displaying more relaxing colors and messages. This system significantly reduces the psychological burden on users during operation.

[1841] The above describes an embodiment of the present invention. This system allows users to intuitively utilize AI functions and efficiently manage tasks and schedules through an interface that adapts to their emotional state during operation.

[1842] The processing flow will be explained below.

[1843] Login process

[1844] Step 1:

[1845] User: Launches the application, enters the username and password on the login screen, and taps the "Login" button.

[1846] Step 2:

[1847] Terminal: Sends an authentication request to the server using the entered username and password.

[1848] Step 3:

[1849] Server: Based on the received request, the server checks the username and password in the database.

[1850] Step 4:

[1851] Server: If authentication is successful, generate an authentication token and send it to the terminal as a response.

[1852] Step 5:

[1853] Device: Receives the authentication token and stores it locally. After storing it, displays the home screen.

[1854] Schedule adjustment operations

[1855] Step 1:

[1856] User: Tap the Schedule button on the home screen.

[1857] Step 2:

[1858] Terminal: Display the schedule adjustment screen.

[1859] Step 3:

[1860] User: Tap and release the event icon (e.g., meeting icon) on the screen and drag it to the desired time slot (e.g., 9:00-10:00).

[1861] Step 4:

[1862] Terminal: Detects the user's tap and release operation and obtains the selected event information and time frame information.

[1863] Step 5:

[1864] Terminal: Creates and sends a request to send the acquired information to the server.

[1865] Step 6:

[1866] Server: Analyzes the received information and registers it in a database.

[1867] Step 7:

[1868] Server: If registration in the database is successful, it sends a success message to the terminal.

[1869] Step 8:

[1870] Terminal: Receives the success message and updates the schedule screen to show the new schedule.

[1871] Task Detailing

[1872] Step 1:

[1873] User: Tap the "Task Management" button on the schedule adjustment screen.

[1874] Step 2:

[1875] Terminal: Display the task management screen.

[1876] Step 3:

[1877] User: Taps a text field to enter new task information.

[1878] Step 4:

[1879] User: Enter task details in the text field and tap the "Done" button.

[1880] Step 5:

[1881] Terminal: Sends the entered task details to the server.

[1882] Step 6:

[1883] Server: Analyzes the received task information and registers it in the database.

[1884] Step 7:

[1885] Server: If registration in the database is successful, it sends a success message to the terminal.

[1886] Step 8:

[1887] Terminal: Receives the success message and updates the task management screen to show the new task information.

[1888] emotion recognition

[1889] Step 1:

[1890] User: During operation, emotions are input through the camera and microphone of the operating device.

[1891] Step 2:

[1892] Terminal: The emotion engine analyzes input data (facial expressions, voice, touch patterns) and recognizes the user's emotions.

[1893] Step 3:

[1894] Terminal: Sends the recognized emotion data to the server.

[1895] Step 4:

[1896] Server: Analyzes the received emotional data and generates instructions to dynamically change the interface display and operation method.

[1897] Step 5:

[1898] Server: Sends instructions to the terminal.

[1899] Step 6:

[1900] Terminal: Receives instructions from the server and dynamically changes the interface display and operation method.

[1901] Specific examples

[1902] For example, consider scheduling a meeting. A user launches an application, enters their login information, and logs in. From the home screen, they tap the "Schedule" button to move to the schedule adjustment screen. They drag the meeting icon to the 9:00 to 10:00 time slot, and the device sends this information to the server. The server registers the information in the database and sends a success message to the device. The device updates the schedule adjustment screen to show the meeting in the 9:00 to 10:00 time slot.

[1903] Furthermore, if the emotion engine detects stress from the user's facial expression during operation, it sends this information to the server. The server then generates instructions for relaxing color tones and message displays and sends them to the device. The device then dynamically changes its interface based on these instructions, improving the user's operating experience.

[1904] Example 2

[1905] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1906] Conventional schedule adjustment and task management systems using operating terminals often cause stress to users because they do not take into account the user's emotional state. Furthermore, when operations are complicated, users have difficulty efficiently managing tasks and schedules. Furthermore, there is a need for a method to smoothly access the system while ensuring the security of authentication information.

[1907] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1908] In this invention, the server includes an operation terminal operated by a user, means for processing input information received from the operation terminal, means for accessing a database stored on the server, means for adjusting schedules and detailing tasks by tapping and releasing on the interface, an emotion engine in the operation terminal that analyzes facial expressions and voice to recognize emotions, and means for transmitting the emotion data to the server and dynamically adjusting the display content and operation method of the interface. This enables dynamic adjustment of the interface according to the user's emotional state, providing a user-friendly operation environment and improving the efficiency of schedule adjustment and task management.

[1909] "User" refers to a person who operates the system via an operation terminal.

[1910] An "operation terminal" is a device operated by a user, and refers to devices such as smartphones and tablets.

[1911] "Server" refers to a computer system that processes input information received from an operating terminal and accesses a database.

[1912] "Database" refers to a data storage system that stores schedule information, task information, authentication information, etc.

[1913] "Interface" refers to a user interface that visually displays events and tasks on a screen displayed on an operating terminal.

[1914] "Tap and release operation" refers to an operation in which a user taps and releases their finger on the touchscreen of the operating device.

[1915] An "emotion engine" refers to a system that analyzes facial expressions and voices on an operating device to recognize emotions.

[1916] "Authentication token" refers to data generated by a server that contains authentication information for an authenticated user.

[1917] "Emotion data" refers to data that indicates the user's emotional state as recognized by the emotion engine.

[1918] System Configuration

[1919] This invention provides a system that combines a tap-and-release operation that allows users to perform operations intuitively with an emotion engine that recognizes the user's emotions. The basic configuration of the system is as follows.

[1920] Operation terminal

[1921] An operating terminal is a device that is directly operated by a user, such as a smartphone or tablet. Examples include iPhones and Android devices. These terminals are equipped with a touch screen, microphone, and camera, and display a user interface.

[1922] server

[1923] The server is a computer system that receives, processes, and stores input information and emotion data sent from the operating device. The server accesses a database to retrieve or register the necessary information. As a specific example, Django is used as the server-side framework.

[1924] Database

[1925] The database is a data storage system stored on a server for storing schedule information, task information, and authentication information. Specific examples include SQLite and MySQL.

[1926] Interface

[1927] The interface is displayed on the operating device and is used by the user with tap and release operations, visually indicating events and tasks and allowing intuitive operation.

[1928] Emotion Engine

[1929] An emotion engine is a system that recognizes emotions by analyzing a user's facial expressions, voice, or touch patterns. For example, OpenCV is used for facial recognition, and TensorFlow is used for emotion analysis. The interface display and operation methods are dynamically changed based on the recognized emotions.

[1930] Usage example

[1931] As a specific example, if the emotion engine detects that a user is feeling stressed while scheduling a meeting, that information is sent from the device to the server. The server can then make adjustments, such as simplifying the interface or displaying more relaxing colors and messages. This system significantly reduces the psychological burden on users during operation.

[1932] Example prompts to input to the generative AI model:

[1933] "How can I recognize a user's emotions while using a scheduling app and adjust the interface when the user feels stressed?"

[1934] The above description concretely illustrates the embodiments of the present invention. This system allows users to intuitively utilize AI functions and efficiently manage tasks and schedules through an interface that adapts to the user's emotional state during operation.

[1935] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1936] Login process

[1937] Step 1:

[1938] User: Launches the application, enters the username and password on the login screen, and taps the "Login" button.

[1939] Input: Username and Password

[1940] Output: Login operation trigger

[1941] Step 2:

[1942] Terminal: Sends an authentication request to the server using the entered username and password.

[1943] Specifically, the terminal generates an HTTP POST request and sends it to the server's authentication API endpoint.

[1944] Input: Username and Password

[1945] Output: Authentication request

[1946] Step 3:

[1947] Server: Based on the received request, the server checks the username and password in the database.

[1948] Specifically, the server runs a database query to verify that the username and hashed password match.

[1949] Input: Authentication Request

[1950] Output: Authentication result

[1951] Step 4:

[1952] Server: If authentication is successful, generate an authentication token and send it to the terminal as a response.

[1953] Specifically, it generates a JWT token along with a success message and returns it as an HTTP response.

[1954] Input: Authentication result

[1955] Output: Authentication token

[1956] Step 5:

[1957] Device: Receives the authentication token and stores it locally. After storing it, displays the home screen.

[1958] Specifically, the token is saved in local storage and the screen transitions to the next screen.

[1959] Input: Authentication Token

[1960] Output: Home screen display

[1961] Schedule adjustment operations

[1962] Step 1:

[1963] User: Tap the Schedule button on the home screen.

[1964] Input: User taps

[1965] Output: Display of schedule screen

[1966] Step 2:

[1967] Terminal: Display the schedule adjustment screen.

[1968] Input: Schedule screen display request

[1969] Output: Schedule adjustment screen

[1970] Step 3:

[1971] User: Tap and release the event icon (e.g., meeting icon) on the screen and drag it to the desired time slot (e.g., 9:00-10:00).

[1972] Input: Tap and release

[1973] Output: Drag operation completed

[1974] Step 4:

[1975] Terminal: Detects the user's tap and release operation and obtains the selected event information and time frame information.

[1976] Specifically, the event ID and time frame are stored as an internal data structure.

[1977] Input: Drag operation complete

[1978] Output: Event information and time slot information

[1979] Step 5:

[1980] Terminal: Creates and sends a request to send the acquired information to the server.

[1981] Specifically, it generates an HTTP POST request and sends it to the server's schedule API endpoint.

[1982] Input: Event information and time slot information

[1983] Output: Reschedule request

[1984] Step 6:

[1985] Server: Analyzes the received information and registers it in a database.

[1986] The specific operation is to execute an INSERT or UPDATE query on the database.

[1987] Input: Schedule change request

[1988] Output: Database update results

[1989] Step 7:

[1990] Server: If registration in the database is successful, it sends a success message to the terminal.

[1991] Specifically, a success status is returned in the HTTP response.

[1992] Input: Database update results

[1993] Output: Success message

[1994] Step 8:

[1995] Terminal: Receives the success message and updates the schedule screen to show the new schedule.

[1996] Specific operations include updating local data and redrawing the user interface.

[1997] Input: Success message

[1998] Output: Updated schedule contents

[1999] Task Detailing

[2000] Step 1:

[2001] User: Tap the "Task Management" button on the schedule adjustment screen.

[2002] Input: User taps

[2003] Output: Display of task management screen

[2004] Step 2:

[2005] Terminal: Display the task management screen.

[2006] Input: Request to display the task management screen

[2007] Output: Task management screen

[2008] Step 3:

[2009] User: Taps a text field to enter new task information.

[2010] Input: Tap operation

[2011] Output: Focus on text field

[2012] Step 4:

[2013] User: Enter task details in the text field and tap the "Done" button.

[2014] Input: Task information

[2015] Output: Task information input completed

[2016] Step 5:

[2017] Terminal: Sends the entered task details to the server.

[2018] Specifically, it generates an HTTP POST request and sends it to the server's task management API endpoint.

[2019] Input: Completed entering task information

[2020] Output: Task add request

[2021] Step 6:

[2022] Server: Analyzes the received task information and registers it in the database.

[2023] As a specific operation, an INSERT query is executed on the database.

[2024] Input: Task Add Request

[2025] Output: Database update results

[2026] Step 7:

[2027] Server: If registration in the database is successful, it sends a success message to the terminal.

[2028] Specifically, a success status is returned in the HTTP response.

[2029] Input: Database update results

[2030] Output: Success message

[2031] Step 8:

[2032] Terminal: Receives the success message and updates the task management screen to show the new task information.

[2033] Specific operations include updating local data and redrawing the user interface.

[2034] Input: Success message

[2035] Output: Updated task information

[2036] emotion recognition

[2037] Step 1:

[2038] User: Facial expressions and voice are input by the emotion engine during operation.

[2039] Input: Facial expressions and voice data

[2040] Output: Input to the emotion engine

[2041] Step 2:

[2042] Terminal: The emotion engine analyzes the input data and recognizes the user's emotions.

[2043] Specifically, it uses machine learning models to classify emotions.

[2044] Input: Facial expressions and voice data

[2045] Output: Recognized emotion data

[2046] Step 3:

[2047] Terminal: Sends the recognized emotion data to the server.

[2048] Specifically, it creates emotion data in JSON format and generates and sends an HTTP POST request.

[2049] Input: Recognized emotion data

[2050] Output: Sending emotion data

[2051] Step 4:

[2052] Server: Receives emotion data and sends instructions back to the device to dynamically adjust the interface display and operation method.

[2053] Specifically, it generates UI change instructions according to the emotion and returns them as an HTTP response.

[2054] Input: Emotion data

[2055] Output: UI change instructions

[2056] Step 5:

[2057] Terminal: Receives instructions from the server and dynamically changes the interface to adapt to the user's current emotional state.

[2058] Specific actions include changing the layout and color of the interface.

[2059] Input: UI change instructions

[2060] Output: Dynamically changed interface

[2061] (Application example 2)

[2062] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2063] Conventional systems lack an operational interface that takes user emotions into account, which means that operability declines when users feel stressed. Furthermore, there is a lack of systems to improve individual customer experiences in brick-and-mortar stores, and there is also the problem that customization suggestions based on user emotions have not yet been fully implemented.

[2064] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2065] In this invention, the server includes an operation terminal operated by a user, a central processing unit that processes input information received from the operation terminal, an information storage device that stores information stored in the central processing unit, a display unit that displays tasks on the operation terminal, a unit that adjusts schedules and subdivides tasks through selection operations on the display unit, an emotion recognition unit that analyzes the user's facial expression and voice data, and a unit that dynamically changes the content of the display unit based on the emotion identified by the emotion recognition unit. This makes it possible to dynamically adjust the interface according to the user's emotion and improve the individual customer experience in physical stores.

[2066] An "operation terminal" is a device that is directly operated by a user, such as a smartphone or tablet.

[2067] The "central processing unit" is a device that processes input information received from an operation terminal, and is usually a computer system called a server.

[2068] "Information storage device" means a device for storing data saved in a central processing unit, and includes a database.

[2069] The "display means" is an interface for displaying tasks and events on the operating terminal, and is generally a touch screen display.

[2070] The "selection operation" refers to an action in which a user selects a task or event on the display means, and includes finger movements such as tapping and swiping.

[2071] "Emotion recognition means" is a technology that identifies emotions by analyzing a user's facial expressions and voice data, and machine learning models are often used.

[2072] The "dynamically changing means" refers to a function of adjusting the content of the display means in real time based on the emotion identified by the emotion recognition means.

[2073] The embodiment of the present invention is a system comprising an operation terminal operated by a user, a central processing unit, an information storage device, a display means, a selection operation means, an emotion recognition means, and a dynamic change means. A specific embodiment of this system will be described in detail below.

[2074] System Configuration

[2075] 1. Operation terminal: A device that is directly operated by the user, such as a smartphone or tablet. It is equipped with a touch screen, camera, and microphone, and can capture the user's facial expressions and voice.

[2076] 2. Central Processing Unit: A computer system that processes input information received from the operating terminal, usually called a server. It integrates multiple input information and retrieves and registers data from the information storage device.

[2077] 3. Information storage device: A device that includes a database and stores schedule information, task information, user authentication information, emotion data, etc.

[2078] 4. Display: An interface displayed on the operating device, providing a visual display to intuitively show tasks and events. It accepts operations such as tapping and swiping.

[2079] 5. Selection operation means: This function detects tap and release operations performed by the user on the display means and allows for schedule adjustments and task subdivision.

[2080] 6. Emotion recognition: Analyzes facial expressions and voice data acquired from the smartphone camera and microphone to identify the user's emotions. Uses machine learning models (e.g., TensorFlow, PyTorch).

[2081] 7. Dynamic modification: This function adjusts the content of the display means in real time based on the emotional data identified by the emotion recognition means. The interface is modified in a way that adapts to the user's current emotional state.

[2082] Program processing explanation

[2083] 1. User: Uses a smartphone to launch the application and operate the interface shown on the display means.

[2084] 2. Device: Detects selection operations such as taps and swipes performed by the user. Also, captures the user's facial expressions and voice using a camera and microphone. This data is sent to the emotion recognition system.

[2085] 3. Emotion recognition means: Analyzes facial expressions and voice data acquired using a machine learning model to identify the user's emotions. The identified emotion data is sent to the central processing unit.

[2086] 4. Central processing unit: After receiving the emotion data, it retrieves related data from the information storage device and generates content to be sent to the display means. It also generates product suggestions and schedule information according to the emotion.

[2087] 5. Display means: Based on the data sent from the central processing unit, the interface content is dynamically changed and displayed to the user, such as special product offers, event information, or relaxing color themes.

[2088] Specific examples

[2089] For example, if a user stops in front of a particular product shelf in a physical store and looks confused, the smartphone's camera and microphone will capture this. The emotion recognition means will identify the emotion as "confusion" and send that data to the central processing unit. The central processing unit will then generate an interface that provides the confused user with detailed product information and usage instructions, and display this on the display means in real time.

[2090] Prompt Sentence Examples

[2091] "Users' emotions are confusing. Can you recommend any products?"

[2092] This system allows the interface to dynamically adjust based on the user's emotions, significantly improving the personalized customer experience in physical stores.

[2093] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2094] Step 1:

[2095] The user starts the application using a smartphone, enters the username and password on the login screen, and taps the "Login" button.

[2096] Input: Username, Password

[2097] Output: Generated authentication request

[2098] Specifically, it detects input on the smartphone's touchscreen and creates an authentication request.

[2099] Step 2:

[2100] The device sends an authentication request to the server.

[2101] Input: Authentication request

[2102] Output: Send data to the server

[2103] Specifically, the network function of the smartphone is used to send authentication request data to the server.

[2104] Step 3:

[2105] Based on the request received by the server, the information storage device checks the user name and password.

[2106] Input: Authentication request (username, password)

[2107] Output: Authentication result (success or failure)

[2108] Specifically, it performs a database query and compares the received data with the stored data.

[2109] Step 4:

[2110] If the server is successful in the authentication, it generates an authentication token and sends it to the terminal as a response.

[2111] Input: Authentication result (success)

[2112] Output: Authentication token

[2113] Specifically, the authentication token is generated using a random number generation algorithm and sent to the terminal.

[2114] Step 5:

[2115] The device receives the authentication token, stores it locally, and displays the home screen.

[2116] Input: Authentication token

[2117] Output: Home screen display

[2118] Specifically, it stores the token in local storage and updates the application's UI component to display the home screen.

[2119] Step 6:

[2120] As the user walks around the store, the device uses a camera and microphone to capture the user's facial expressions and voice.

[2121] Input: Camera video, audio data

[2122] Output: Captured data

[2123] Specifically, it periodically collects data from the camera and microphone sensors.

[2124] Step 7:

[2125] An emotion recognition means in the terminal analyzes the captured data and identifies the user's emotion.

[2126] Input: Captured data (facial expression, voice)

[2127] Output: Recognized emotion data

[2128] Specifically, it uses machine learning models to analyze data and identify specific emotions.

[2129] Step 8:

[2130] The device transmits the recognized emotion data to the server.

[2131] Input: Emotion data

[2132] Output: Send emotion data to the server

[2133] As a specific operation, emotion data is transmitted via a network.

[2134] Step 9:

[2135] The server receives the emotion data and uses a generative AI model to generate data for appropriate product suggestions and interface changes.

[2136] Input: Emotion data

[2137] Output: Proposal data, interface change data

[2138] Specifically, the prompt sentence "The user is confused. Please recommend some products." is input into the AI ​​model, and the generated suggestions are obtained.

[2139] Step 10:

[2140] The server transmits the generated proposal data to the terminal.

[2141] Input: Proposal data

[2142] Output: Sending data to the terminal

[2143] As a specific operation, the proposal data is transmitted to the terminal via the network.

[2144] Step 11:

[2145] The terminal receives the proposal data and dynamically adjusts and presents an interface to the user using the display means.

[2146] Input: Proposal data

[2147] Output: Dynamically adjusted interface

[2148] Specifically, it updates interface elements such as display content, color themes, and messages and displays them to the user.

[2149] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2150] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2151] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2152] [Fourth embodiment]

[2153] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2154] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2155] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2156] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2157] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2158] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2159] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2160] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2161] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2162] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2163] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2164] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2165] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2166] The purpose of this invention is to utilize an interface that users can operate intuitively to more effectively and easily utilize the functions of artificial intelligence (AI). A specific embodiment of this system and its operation will be described below.

[2167] System Configuration

[2168] The system mainly consists of the following components:

[2169] 1. Operation terminal

[2170] A device that is directly operated by the user. Examples include smartphones and tablet devices.

[2171] It has a touch screen that displays the interface.

[2172] 2. Server

[2173] It receives input information sent from the operating terminal, processes it, and stores it.

[2174] Access the database and retrieve or register the required information.

[2175] 3. Database

[2176] It is stored on the server and stores schedule information, task information, etc.

[2177] 4. Interface

[2178] It is displayed on the operating device and is used by the user with a tap and release operation.

[2179] Events and tasks are visually displayed, allowing for intuitive operation.

[2180] Program processing

[2181] The following describes the program processing in this system. The processing of each function is described with the subject being either the operating terminal, the server, or the user.

[2182] Login process

[2183] 1. User: Launches the application and enters username and password on the login screen.

[2184] 2. Terminal: Sends the entered username and password to the server.

[2185] 3. Server: Validates the username and password against the database and, if authentication is successful, generates an authentication token.

[2186] 4. Server: Sends the authentication token to the operating terminal.

[2187] 5. Device: Stores the received authentication token locally and displays the user's home screen.

[2188] Schedule adjustment operations

[2189] 1. User: Tap the "Schedule" button from the home screen.

[2190] 2. Terminal: Displays the schedule adjustment screen and allows the user to select an event.

[2191] 3. User: Tap and release the event icon on the interface and drag it to the desired time frame.

[2192] 4. Terminal: Obtains the information of the selected event and time frame and sends it to the server.

[2193] 5. Server: Registers the received information in a database and sends the processing results to the terminal.

[2194] 6. Terminal: Receives the processing results, updates the schedule screen, and displays the registered details.

[2195] Task Detailing

[2196] 1. User: Tap the "Task Management" button on the schedule adjustment screen.

[2197] 2. Terminal: Displays the task management screen and provides text fields for entering details.

[2198] 3. User: Enter task details in the text field and tap the "Done" button.

[2199] 4. Terminal: Sends the entered task information to the server.

[2200] 5. Server: Registers the received information in a database and sends the processing results to the terminal.

[2201] 6. Terminal: Receives the processing results and updates the task management screen to display the new task.

[2202] Specific examples

[2203] For example, consider a user who wants to schedule a meeting from 9:00 to 10:00. The user first launches the application and enters their login information. After logging in, they navigate to the home screen and tap the "Schedule" button. The schedule adjustment screen appears, and the user drags the meeting icon to the 9:00 to 10:00 time slot. This operation causes the device to send the event information and time slot information to the server, which then registers it in the database. As a result, the end user can see that the schedule has been updated appropriately.

[2204] The above describes an embodiment of the present invention. This system allows users to intuitively utilize AI functions and efficiently adjust schedules and refine tasks.

[2205] The processing flow will be explained below.

[2206] Login process

[2207] Step 1:

[2208] User: Launches the application, enters the username and password on the login screen, and taps the "Login" button.

[2209] Step 2:

[2210] Terminal: Sends an authentication request to the server using the entered username and password.

[2211] Step 3:

[2212] Server: Based on the received request, the server checks the username and password in the database.

[2213] Step 4:

[2214] Server: If authentication is successful, generate an authentication token and send it to the terminal as a response.

[2215] Step 5:

[2216] Device: Receives the authentication token and stores it locally. After storing it, displays the home screen.

[2217] Schedule adjustment operations

[2218] Step 1:

[2219] User: Tap the Schedule button on the home screen.

[2220] Step 2:

[2221] Terminal: Display the schedule adjustment screen.

[2222] Step 3:

[2223] User: Tap and release the event icon (e.g., meeting icon) on the screen and drag it to the desired time slot (e.g., 9:00-10:00).

[2224] Step 4:

[2225] Terminal: Detects the user's tap and release operation and obtains the selected event information and time frame information.

[2226] Step 5:

[2227] Terminal: Creates and sends a request to send the acquired information to the server.

[2228] Step 6:

[2229] Server: Analyzes the received information and registers it in a database.

[2230] Step 7:

[2231] Server: If registration in the database is successful, it sends a success message to the terminal.

[2232] Step 8:

[2233] Terminal: Receives the success message and updates the schedule screen to show the new schedule.

[2234] Task Detailing

[2235] Step 1:

[2236] User: Tap the "Task Management" button on the schedule adjustment screen.

[2237] Step 2:

[2238] Terminal: Display the task management screen.

[2239] Step 3:

[2240] User: Taps a text field to enter new task information.

[2241] Step 4:

[2242] User: Enter task details in the text field and tap the "Done" button.

[2243] Step 5:

[2244] Terminal: Sends the entered task details to the server.

[2245] Step 6:

[2246] Server: Analyzes the received task information and registers it in the database.

[2247] Step 7:

[2248] Server: If registration in the database is successful, it sends a success message to the terminal.

[2249] Step 8:

[2250] Terminal: Receives the success message and updates the task management screen to show the new task information.

[2251] Example 1

[2252] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2253] In conventional systems, it was difficult for users to efficiently adjust schedules and detail tasks using an intuitive interface. The authentication process was also cumbersome, resulting in a poor user experience. Furthermore, there was a lack of a mechanism for reliably and securely transmitting task and event information to the server.

[2254] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2255] In this invention, the server includes means for processing input information received from the operation terminal, a database stored in the operation terminal, an interface means displayed on the operation terminal for displaying events and tasks, means for adjusting schedules and refining tasks through tap-and-release operations on the interface, means for verifying authentication information received from the operation terminal, generating an authentication token and transmitting it to the operation terminal, and means for transmitting event information and time slot information acquired based on the tap-and-release operations to the server. This allows users to effectively utilize AI functions through intuitive operations, enabling efficient schedule adjustments and task refinement.

[2256] An "information processing terminal" is a device that is operated by a user to input and display information.

[2257] The "arithmetic unit" is a device that processes input information received from an information processing terminal, and performs database management and various other operations.

[2258] An "information storage device" is a storage medium connected to a computing device for storing and managing data.

[2259] The "display means" is an interface that is built into the information processing terminal and that visually shows the work contents and schedule to the user.

[2260] The term "instruction operation" refers to an operation performed with a finger, a stylus, or the like using the display means of the information processing terminal.

[2261] "Authentication information" refers to information such as a user name and password that a user enters when using an information processing terminal.

[2262] An "authentication token" is data that indicates the authentication of a user and is generated when authentication information is successfully verified.

[2263] "Work information" is detailed information about tasks and events.

[2264] "Time information" is information about the time period in which a task or event is executed.

[2265] This invention relates to a system for effectively and easily adjusting schedules and detailing tasks via an intuitive interface on an information processing terminal (e.g., a smartphone or tablet) operated by a user.

[2266] System Configuration

[2267] The system mainly consists of the following components:

[2268] 1. Information processing terminal

[2269] It is a device that is directly operated by the user.

[2270] Examples include smartphones and tablet devices, which have touchscreens.

[2271] This terminal is equipped with a display means for visually showing the work content and schedule.

[2272] 2. Arithmetic device

[2273] It processes input information received from the information processing terminal and performs database management and various operations.

[2274] This applies to web servers or cloud servers.

[2275] 3. Information storage device

[2276] It is a storage medium connected to a computing device for storing and managing data.

[2277] A database management system (RDBMS) is used.

[2278] System Functions and Operation

[2279] Examples of the main functions of this system include login processing, schedule adjustment, and task detailing. The operation of each function is as follows:

[2280] Login process

[2281] 1. The user launches the application and enters authentication information (username and password) on the login screen.

[2282] 2. The information processing terminal transmits the input authentication information to the arithmetic unit.

[2283] 3. The computing device checks the authentication information against information stored in a database and, if successful, generates an authentication token.

[2284] 4. The computing device transmits the authentication token to the information processing terminal.

[2285] 5. The information processing terminal stores the authentication token in local storage and displays the user's home screen.

[2286] Schedule adjustment

[2287] 1. The user taps the "Schedule" button on the home screen.

[2288] 2. The information processing terminal displays the schedule screen, and the user drags and drops the schedule into the desired time slot.

[2289] 3. The information processing terminal transmits information about the selected schedule and time slot to the computing device.

[2290] 4. The arithmetic unit registers the received information in a database and returns the processing results to the information processing terminal.

[2291] 5. The information processing terminal receives the processing results and updates the schedule screen.

[2292] Task Detailing

[2293] 1. The user taps the "Task Management" button on the schedule adjustment screen.

[2294] 2. The information processing terminal displays the task management screen, and the user enters the task details.

[2295] 3. The information processing terminal transmits the input task information to the arithmetic unit.

[2296] 4. The arithmetic unit registers the received information in a database and returns the processing results to the information processing terminal.

[2297] 5. The information processing terminal receives the processing results and updates the task management screen.

[2298] Specific examples

[2299] For example, if a user "reschedules a meeting from 9 to 10":

[2300] 1. The user launches the application and enters their username and password at the login screen.

[2301] 2. The user taps the Schedule button on the home screen.

[2302] 3. The information processing terminal displays the schedule screen.

[2303] 4. The user drags and drops the meeting icon into the 9:00 to 10:00 time slot.

[2304] 5. The information processing terminal transmits the information about the selected meeting and the new time slot information to the computing device.

[2305] 6. The arithmetic unit registers the received data in a database and returns the processing results to the information processing terminal.

[2306] 7. The information processing terminal receives the processing results and displays that the schedule has been appropriately updated.

[2307] Example prompts for generative AI models

[2308] "Generate a sentence that explains the steps a user takes to reschedule a meeting from 9:00 to 10:00. The steps include launching the application, logging in, displaying the schedule screen, selecting the meeting icon, moving to the time slot, sending the data to the server, registering it in the database, and checking for updates."

[2309] This system allows users to intuitively utilize AI functions to efficiently adjust schedules and detail tasks.

[2310] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2311] Login process

[2312] Step 1:

[2313] A user launches an application and is directed to a login screen, where they enter their authentication information (username and password).

[2314] Input: Username and Password

[2315] Output: Create login request

[2316] Step 2:

[2317] The device sends the entered username and password to the server using the HTTPS protocol, which prevents unauthorized interception of the data.

[2318] Input: Username and Password

[2319] Output: HTTPS request

[2320] Step 3:

[2321] The server checks the received username and password against the information stored in the database, using an SQL query to look up the user information and confirm a match.

[2322] Input: Username and password in the HTTPS request

[2323] Output: Authentication result (success or failure)

[2324] Step 4:

[2325] If the server is successful in authenticating, it generates an authentication token, such as a JWT (JSON Web Token), which is used to ensure that subsequent requests are valid.

[2326] Input: Authentication result (success)

[2327] Output: Authentication token

[2328] Step 5:

[2329] The server sends the generated authentication token to the terminal using the HTTPS protocol.

[2330] Input: Authentication Token

[2331] Output: HTTPS response

[2332] Step 6:

[2333] The device stores the received authentication token in local storage and prepares to display the user's home screen.

[2334] Input: HTTPS response containing an authentication token

[2335] Output: Home screen display

[2336] Schedule adjustment operations

[2337] Step 1:

[2338] The user taps the "Schedule" button from the home screen, which takes the user to the schedule adjustment screen.

[2339] Input: Home screen interaction

[2340] Output: Schedule screen request

[2341] Step 2:

[2342] The device displays a schedule adjustment screen, providing a calendar and time slot interface.

[2343] Input: Schedule screen request

[2344] Output: Schedule adjustment screen

[2345] Step 3:

[2346] Users can visually change the schedule by dragging and dropping event icons onto the desired time slots on the interface.

[2347] Input: Drag and drop of event icons

[2348] Output: New timeframe selection information

[2349] Step 4:

[2350] The device sends the selected event information and new time slot information to the server in JSON format, which is used for subsequent processing.

[2351] Input: New timeframe selection information

[2352] Output: JSON format data

[2353] Step 5:

[2354] The server parses the received JSON data and executes SQL queries to register the relationship between events and time frames in the database.

[2355] Input: JSON format data

[2356] Output: Registration results in the database

[2357] Step 6:

[2358] The server returns the processing result (success or failure) to the terminal along with a status code.

[2359] Input: Registration results to the database

[2360] Output: Response with status code

[2361] Step 7:

[2362] The device receives the status code and updates the schedule screen accordingly. If successful, the new schedule information is displayed.

[2363] Input: Response with status code

[2364] Output: Updated schedule screen

[2365] Task Detailing

[2366] Step 1:

[2367] The user taps the "Task Management" button on the schedule adjustment screen, which takes the user to the task management screen.

[2368] Input: Schedule adjustment screen interaction

[2369] Output: Task management screen request

[2370] Step 2:

[2371] The device displays a task management screen, providing text fields and options for the user to enter task details.

[2372] Input: Task management screen request

[2373] Output: Task management screen

[2374] Step 3:

[2375] The user fills in the task details in the input fields and taps the "Done" button, which creates a new task.

[2376] Input: Enter task details

[2377] Output: New task information

[2378] Step 4:

[2379] The device sends new task information in JSON format to the server.

[2380] Input: New task information

[2381] Output: JSON format data

[2382] Step 5:

[2383] The server parses the received JSON data and executes an SQL query to register new task information in the database.

[2384] Input: JSON format data

[2385] Output: Registration results in the database

[2386] Step 6:

[2387] The server returns the processing result (success or failure) to the terminal along with a status code.

[2388] Input: Registration results to the database

[2389] Output: Response with status code

[2390] Step 7:

[2391] The device receives the status code and updates the task management screen accordingly, displaying the new task information if successful.

[2392] Input: Response with status code

[2393] Output: Updated task management screen

[2394] (Application example 1)

[2395] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2396] Managing robot operation schedules and task allocation within a factory is complex and time-consuming, placing a significant burden on managers. This burden increases even more when real-time adjustments and changes are required. Current systems lack an intuitive interface for performing these operations, making efficient management difficult. Therefore, there is a need for a method for efficiently managing robot operation schedules and task allocation within a factory using an intuitive interface.

[2397] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2398] In this invention, the server includes an operation terminal operated by a user, means for processing input information received from the operation terminal, a database stored in the server, an interface that indicates events and tasks and is displayed on the operation terminal, means for adjusting schedules and refining tasks by tapping and releasing on the interface, and means for intuitively managing robot operation schedules and task allocation within a factory via the operation terminal. This enables a user to efficiently manage robot operation schedules and task allocation within a factory using an interface that can be intuitively operated.

[2399] A "user" is a person who operates the system using an operation terminal.

[2400] An "operation terminal" is a device used by a user to adjust schedules and manage tasks, and includes smartphones, tablet terminals, and the like.

[2401] A "server" is a device that processes input information sent from an operating terminal and manages data stored in a database.

[2402] A "database" is a system that is stored on a server and stores data such as schedule information and task information.

[2403] An "interface" is a screen that is displayed on an operation terminal and allows a user to visually check events and tasks and perform operations.

[2404] A "tap and release operation" is a series of operations in which a user taps, drags, and then releases their finger on the touch screen of the operating terminal.

[2405] "Schedule adjustment" refers to an operation in which a user sets or changes the time frame of an event or task on the interface.

[2406] "Task elaboration" is an operation in which a user inputs and manages detailed information about a specific task.

[2407] A "robot" is an automated mechanical device used in factories that has the ability to perform various tasks.

[2408] A "robot operation schedule" is a plan that indicates what operations a robot will perform at what time periods.

[2409] "Task allocation" means assigning specific tasks to each robot.

[2410] This invention provides a system that allows intuitive management of robot operation schedules and task allocation within a factory. The main elements of the system are an operation terminal, a server, a database, and an interface. Specific mechanisms include login processing, schedule adjustment, and task detailing.

[2411] System Configuration

[2412] The system consists of the following elements:

[2413] 1. Operation terminal:

[2414] A device operated by a user, such as a smartphone or tablet.

[2415] It is equipped with a touch screen that allows users to operate it intuitively.

[2416] 2. Server:

[2417] This is a device that receives, processes, and saves input information sent from an operating terminal.

[2418] Access a database to manage schedule and task-related information.

[2419] 3. Database:

[2420] It is a system that stores schedule information, task information, etc. on a server.

[2421] 4. Interface:

[2422] This screen is displayed on the operation terminal and allows the user to visually check and operate events and tasks.

[2423] You can adjust schedules and detail tasks with a tap and release operation.

[2424] Program processing

[2425] Hardware and Software

[2426] Hardware:

[2427] Smartphone: A device running an operating system such as iOS or Android.

[2428] Server: A device that processes data and manages databases.

[2429] software:

[2430] Flask: A Python web framework for running server-side programs.

[2431] SQLite: A lightweight relational database management system.

[2432] Data processing and calculation

[2433] The server processes the authentication information received from the operation terminal and generates an authentication token. It also processes the event information and time slot information received from the operation terminal and registers it in a database. Specifically, when a user adjusts a schedule or details a task on the operation terminal, the operation terminal sends this data to the server. The server processes the received data, stores it in the database, and returns the results to the operation terminal. This allows users to intuitively manage the operation schedules and task allocation of robots in the factory.

[2434] Specific examples

[2435] For example, if a factory manager sets the operation schedule for Robot A from 1:00 to 2:00 and adds the task "inspection work," the process is as follows:

[2436] 1. Launch the app and log in by entering your username and password.

[2437] 2. Tap the "Schedule" button on the home screen and drag the Robot A icon to the 1:00 to 2:00 time slot.

[2438] 3. Next, tap the "Task Management" button, enter "Inspection Work," and tap "Done."

[2439] Prompt Sentence Examples

[2440] An example prompt is:

[2441] "Please generate a program to schedule Robot A from 1:00 to 2:00 and add 'Inspection Work' to the task."

[2442] This allows users to easily manage schedules and tasks for robots in their factories, improving factory operational efficiency.

[2443] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2444] Step 1:

[2445] The user starts up the operation terminal and enters the user name and password on the login screen. This input information is sent from the operation terminal to the server.

[2446] Input: Username, Password

[2447] Output: Authentication token, login success / failure message

[2448] Specific actions: The user uses the touchscreen to enter login information and taps the submit button.

[2449] Step 2:

[2450] The server verifies the received username and password in the database. If authentication is successful, it generates an authentication token and sends it to the operating terminal.

[2451] Input: Username, Password

[2452] Output: Authentication token

[2453] Specific operation: The server queries the database to verify the user information. If authentication is successful, the server generates an authentication token and sends it to the operating terminal.

[2454] Step 3:

[2455] The operation terminal locally stores the received authentication token and displays the user's home screen.

[2456] Input: Authentication Token

[2457] Output: Home screen

[2458] Specific operation: The operating terminal stores the authentication token and displays the home screen.

[2459] Step 4:

[2460] The user taps the "Schedule" button on the home screen to display the schedule adjustment screen.

[2461] Input: Tap operation

[2462] Output: Schedule adjustment screen

[2463] Specific actions: The user taps the "Schedule" icon on the home screen.

[2464] Step 5:

[2465] The user drags and drops a specific robot icon into the desired time slot on the schedule adjustment screen. The operating terminal acquires event information and time slot information based on this operation.

[2466] Input: Tap and release, drag

[2467] Output: Event information, time slot information

[2468] Specific behavior: The user touches the robot icon, drags it to the desired time frame, and releases it.

[2469] Step 6:

[2470] The operation terminal sends the acquired event information and time slot information to the server, which stores the received information in a database and updates the schedule based on that information.

[2471] Input: Event information, time slot information

[2472] Output: Schedule update information

[2473] Specific operation: The operation terminal sends the event information and time frame information to the server, and the server executes an update query on the database.

[2474] Step 7:

[2475] The server transmits the processing result to the operation terminal, and the operation terminal updates the schedule adjustment screen to display the new schedule.

[2476] Input: Schedule Update Information

[2477] Output: Updated schedule screen

[2478] Specific operation: The server sends schedule update information to the operation terminal, and the operation terminal updates the schedule screen.

[2479] Step 8:

[2480] The user taps the "Task Management" button on the schedule adjustment screen to display the task management screen.

[2481] Input: Tap operation

[2482] Output: Task management screen

[2483] Specific action: The user taps the "Task Management" button.

[2484] Step 9:

[2485] The user enters the details of the task to be assigned to a specific robot on the task management screen and taps the "Done" button. The operating device then sends this information to the server.

[2486] Input: Task details

[2487] Output: Task registration information

[2488] Specific actions: The user enters task details and taps the "Done" button.

[2489] Step 10:

[2490] The server stores the received task detail information in a database and transmits the processing results to the operating terminal.

[2491] Input: Task details

[2492] Output: Task registration result

[2493] Specific operation: The server saves the task details in the database and sends the processing results to the operating terminal.

[2494] Step 11:

[2495] The operation terminal updates the task management screen to display the new task.

[2496] Input: Task registration result

[2497] Output: Updated task management screen

[2498] Specific operation: The operation terminal updates the task management screen based on the task registration results and displays the new task.

[2499] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2500] System Configuration

[2501] This invention provides a system that combines a tap-and-release operation that allows users to perform operations intuitively with an emotion engine that recognizes the user's emotions. Specific embodiments of this system and their operation will be described below.

[2502] Basic Configuration

[2503] 1. Operation terminal:

[2504] These are devices that are directly operated by the user, such as smartphones and tablets.

[2505] It has a touch screen that displays the interface.

[2506] It also has a microphone and a camera for capturing the user's voice and facial expressions.

[2507] 2. Server:

[2508] It receives input information and emotion data sent from the operating terminal, processes them, and stores them.

[2509] Access the database and retrieve or register the required information.

[2510] 3. Database:

[2511] It is stored on the server and contains schedule information, task information, and authentication information.

[2512] 4. Interface:

[2513] It is displayed on the operating device and is used by the user with a tap and release operation.

[2514] Events and tasks are visually displayed, allowing for intuitive operation.

[2515] 5. Emotion Engine:

[2516] It analyzes the user's facial expressions, voice, or touch patterns to recognize emotions.

[2517] The interface display and operation methods are dynamically changed based on the recognized emotions.

[2518] Program processing

[2519] The program processing in this system will be explained below.

[2520] Login process

[2521] 1. User: Launch the application, enter your username and password on the login screen, and tap the "Login" button.

[2522] 2. Terminal: Sends an authentication request to the server using the entered username and password.

[2523] 3. Server: Based on the received request, it checks the username and password in the database.

[2524] 4. Server: If authentication is successful, generate an authentication token and send it to the terminal as a response.

[2525] 5. Device: Receives the authentication token and saves it locally. After saving, displays the home screen.

[2526] Schedule adjustment operations

[2527] 1. User: Tap the "Schedule" button on the home screen.

[2528] 2. Device: Display the schedule adjustment screen.

[2529] 3. User: Tap and release the event icon (e.g., meeting icon) on the screen and drag it to the desired time slot (e.g., 9:00-10:00).

[2530] 4. Terminal: Detects the user's tap and release operation and obtains the selected event information and time frame information.

[2531] 5. Terminal: Creates and sends a request to send the acquired information to the server.

[2532] 6. Server: Analyzes the received information and registers it in a database.

[2533] 7. Server: If registration in the database is successful, it sends a success message to the terminal.

[2534] 8. Terminal: Receives the success message and updates the schedule screen to show the new schedule.

[2535] Task Detailing

[2536] 1. User: Tap the "Task Management" button on the schedule adjustment screen.

[2537] 2. Device: Display the task management screen.

[2538] 3. User: Taps the text field to enter new task information.

[2539] 4. User: Enter task details in the text field and tap the "Done" button.

[2540] 5. Terminal: Sends the entered task details to the server.

[2541] 6. Server: Analyzes the received task information and registers it in the database.

[2542] 7. Server: If registration in the database is successful, it sends a success message to the terminal.

[2543] 8. Terminal: Receives the success message and updates the task management screen to show the new task information.

[2544] emotion recognition

[2545] 1. User: Facial expressions and voice are input by the emotion engine during operation.

[2546] 2. Terminal: The emotion engine analyzes the input data and recognizes the user's emotions.

[2547] 3. Terminal: Sends the recognized emotion data to the server.

[2548] 4. Server: Receives emotion data and sends instructions back to the device to dynamically adjust the interface display and operation method.

[2549] 5. Terminal: Dynamically changes the interface based on instructions from the server to adapt to the user's current emotional state.

[2550] Specific examples

[2551] For example, if the emotion engine detects that a user is feeling stressed while scheduling a meeting, that information is sent from the device to the server. The server can then make adjustments, such as simplifying the interface or displaying more relaxing colors and messages. This system significantly reduces the psychological burden on users during operation.

[2552] The above describes an embodiment of the present invention. This system allows users to intuitively utilize AI functions and efficiently manage tasks and schedules through an interface that adapts to their emotional state during operation.

[2553] The processing flow will be explained below.

[2554] Login process

[2555] Step 1:

[2556] User: Launches the application, enters the username and password on the login screen, and taps the "Login" button.

[2557] Step 2:

[2558] Terminal: Sends an authentication request to the server using the entered username and password.

[2559] Step 3:

[2560] Server: Based on the received request, the server checks the username and password in the database.

[2561] Step 4:

[2562] Server: If authentication is successful, generate an authentication token and send it to the terminal as a response.

[2563] Step 5:

[2564] Device: Receives the authentication token and stores it locally. After storing it, displays the home screen.

[2565] Schedule adjustment operations

[2566] Step 1:

[2567] User: Tap the Schedule button on the home screen.

[2568] Step 2:

[2569] Terminal: Display the schedule adjustment screen.

[2570] Step 3:

[2571] User: Tap and release the event icon (e.g., meeting icon) on the screen and drag it to the desired time slot (e.g., 9:00-10:00).

[2572] Step 4:

[2573] Terminal: Detects the user's tap and release operation and obtains the selected event information and time frame information.

[2574] Step 5:

[2575] Terminal: Creates and sends a request to send the acquired information to the server.

[2576] Step 6:

[2577] Server: Analyzes the received information and registers it in a database.

[2578] Step 7:

[2579] Server: If registration in the database is successful, it sends a success message to the terminal.

[2580] Step 8:

[2581] Terminal: Receives the success message and updates the schedule screen to show the new schedule.

[2582] Task Detailing

[2583] Step 1:

[2584] User: Tap the "Task Management" button on the schedule adjustment screen.

[2585] Step 2:

[2586] Terminal: Display the task management screen.

[2587] Step 3:

[2588] User: Taps a text field to enter new task information.

[2589] Step 4:

[2590] User: Enter task details in the text field and tap the "Done" button.

[2591] Step 5:

[2592] Terminal: Sends the entered task details to the server.

[2593] Step 6:

[2594] Server: Analyzes the received task information and registers it in the database.

[2595] Step 7:

[2596] Server: If registration in the database is successful, it sends a success message to the terminal.

[2597] Step 8:

[2598] Terminal: Receives the success message and updates the task management screen to show the new task information.

[2599] emotion recognition

[2600] Step 1:

[2601] User: During operation, emotions are input through the camera and microphone of the operating device.

[2602] Step 2:

[2603] Terminal: The emotion engine analyzes input data (facial expressions, voice, touch patterns) and recognizes the user's emotions.

[2604] Step 3:

[2605] Terminal: Sends the recognized emotion data to the server.

[2606] Step 4:

[2607] Server: Analyzes the received emotional data and generates instructions to dynamically change the interface display and operation method.

[2608] Step 5:

[2609] Server: Sends instructions to the terminal.

[2610] Step 6:

[2611] Terminal: Receives instructions from the server and dynamically changes the interface display and operation method.

[2612] Specific examples

[2613] For example, consider scheduling a meeting. A user launches an application, enters their login information, and logs in. From the home screen, they tap the "Schedule" button to move to the schedule adjustment screen. They drag the meeting icon to the 9:00 to 10:00 time slot, and the device sends this information to the server. The server registers the information in the database and sends a success message to the device. The device updates the schedule adjustment screen to show the meeting in the 9:00 to 10:00 time slot.

[2614] Furthermore, if the emotion engine detects stress from the user's facial expression during operation, it sends this information to the server. The server then generates instructions for relaxing color tones and message displays and sends them to the device. The device then dynamically changes its interface based on these instructions, improving the user's operating experience.

[2615] Example 2

[2616] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2617] Conventional schedule adjustment and task management systems using operating terminals often cause stress to users because they do not take into account the user's emotional state. Furthermore, when operations are complicated, users have difficulty efficiently managing tasks and schedules. Furthermore, there is a need for a method to smoothly access the system while ensuring the security of authentication information.

[2618] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2619] In this invention, the server includes an operation terminal operated by a user, means for processing input information received from the operation terminal, means for accessing a database stored on the server, means for adjusting schedules and detailing tasks by tapping and releasing on the interface, an emotion engine in the operation terminal that analyzes facial expressions and voice to recognize emotions, and means for transmitting the emotion data to the server and dynamically adjusting the display content and operation method of the interface. This enables dynamic adjustment of the interface according to the user's emotional state, providing a user-friendly operation environment and improving the efficiency of schedule adjustment and task management.

[2620] "User" refers to a person who operates the system via an operation terminal.

[2621] An "operation terminal" is a device operated by a user, and refers to devices such as smartphones and tablets.

[2622] "Server" refers to a computer system that processes input information received from an operating terminal and accesses a database.

[2623] "Database" refers to a data storage system that stores schedule information, task information, authentication information, etc.

[2624] "Interface" refers to a user interface that visually displays events and tasks on a screen displayed on an operating terminal.

[2625] "Tap and release operation" refers to an operation in which a user taps and releases their finger on the touchscreen of the operating device.

[2626] An "emotion engine" refers to a system that analyzes facial expressions and voices on an operating device to recognize emotions.

[2627] "Authentication token" refers to data generated by a server that contains authentication information for an authenticated user.

[2628] "Emotion data" refers to data that indicates the user's emotional state as recognized by the emotion engine.

[2629] System Configuration

[2630] This invention provides a system that combines a tap-and-release operation that allows users to perform operations intuitively with an emotion engine that recognizes the user's emotions. The basic configuration of the system is as follows.

[2631] Operation terminal

[2632] An operating terminal is a device that is directly operated by a user, such as a smartphone or tablet. Examples include iPhones and Android devices. These terminals are equipped with a touch screen, microphone, and camera, and display a user interface.

[2633] server

[2634] The server is a computer system that receives, processes, and stores input information and emotion data sent from the operating device. The server accesses a database to retrieve or register the necessary information. As a specific example, Django is used as the server-side framework.

[2635] Database

[2636] The database is a data storage system stored on a server for storing schedule information, task information, and authentication information. Specific examples include SQLite and MySQL.

[2637] Interface

[2638] The interface is displayed on the operating device and is used by the user with tap and release operations, visually indicating events and tasks and allowing intuitive operation.

[2639] Emotion Engine

[2640] An emotion engine is a system that recognizes emotions by analyzing a user's facial expressions, voice, or touch patterns. For example, OpenCV is used for facial recognition, and TensorFlow is used for emotion analysis. The interface display and operation methods are dynamically changed based on the recognized emotions.

[2641] Usage example

[2642] As a specific example, if the emotion engine detects that a user is feeling stressed while scheduling a meeting, that information is sent from the device to the server. The server can then make adjustments, such as simplifying the interface or displaying more relaxing colors and messages. This system significantly reduces the psychological burden on users during operation.

[2643] Example prompts to input to the generative AI model:

[2644] "How can I recognize a user's emotions while using a scheduling app and adjust the interface when the user feels stressed?"

[2645] The above description concretely illustrates the embodiments of the present invention. This system allows users to intuitively utilize AI functions and efficiently manage tasks and schedules through an interface that adapts to the user's emotional state during operation.

[2646] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2647] Login process

[2648] Step 1:

[2649] User: Launches the application, enters the username and password on the login screen, and taps the "Login" button.

[2650] Input: Username and Password

[2651] Output: Login operation trigger

[2652] Step 2:

[2653] Terminal: Sends an authentication request to the server using the entered username and password.

[2654] Specifically, the terminal generates an HTTP POST request and sends it to the server's authentication API endpoint.

[2655] Input: Username and Password

[2656] Output: Authentication request

[2657] Step 3:

[2658] Server: Based on the received request, the server checks the username and password in the database.

[2659] Specifically, the server runs a database query to verify that the username and hashed password match.

[2660] Input: Authentication Request

[2661] Output: Authentication result

[2662] Step 4:

[2663] Server: If authentication is successful, generate an authentication token and send it to the terminal as a response.

[2664] Specifically, it generates a JWT token along with a success message and returns it as an HTTP response.

[2665] Input: Authentication result

[2666] Output: Authentication token

[2667] Step 5:

[2668] Device: Receives the authentication token and stores it locally. After storing it, displays the home screen.

[2669] Specifically, the token is saved in local storage and the screen transitions to the next screen.

[2670] Input: Authentication Token

[2671] Output: Home screen display

[2672] Schedule adjustment operations

[2673] Step 1:

[2674] User: Tap the Schedule button on the home screen.

[2675] Input: User taps

[2676] Output: Display of schedule screen

[2677] Step 2:

[2678] Terminal: Display the schedule adjustment screen.

[2679] Input: Schedule screen display request

[2680] Output: Schedule adjustment screen

[2681] Step 3:

[2682] User: Tap and release the event icon (e.g., meeting icon) on the screen and drag it to the desired time slot (e.g., 9:00-10:00).

[2683] Input: Tap and release

[2684] Output: Drag operation completed

[2685] Step 4:

[2686] Terminal: Detects the user's tap and release operation and obtains the selected event information and time frame information.

[2687] Specifically, the event ID and time frame are stored as an internal data structure.

[2688] Input: Drag operation complete

[2689] Output: Event information and time slot information

[2690] Step 5:

[2691] Terminal: Creates and sends a request to send the acquired information to the server.

[2692] Specifically, it generates an HTTP POST request and sends it to the server's schedule API endpoint.

[2693] Input: Event information and time slot information

[2694] Output: Reschedule request

[2695] Step 6:

[2696] Server: Analyzes the received information and registers it in a database.

[2697] The specific operation is to execute an INSERT or UPDATE query on the database.

[2698] Input: Schedule change request

[2699] Output: Database update results

[2700] Step 7:

[2701] Server: If registration in the database is successful, it sends a success message to the terminal.

[2702] Specifically, a success status is returned in the HTTP response.

[2703] Input: Database update results

[2704] Output: Success message

[2705] Step 8:

[2706] Terminal: Receives the success message and updates the schedule screen to show the new schedule.

[2707] Specific operations include updating local data and redrawing the user interface.

[2708] Input: Success message

[2709] Output: Updated schedule contents

[2710] Task Detailing

[2711] Step 1:

[2712] User: Tap the "Task Management" button on the schedule adjustment screen.

[2713] Input: User taps

[2714] Output: Display of task management screen

[2715] Step 2:

[2716] Terminal: Display the task management screen.

[2717] Input: Request to display the task management screen

[2718] Output: Task management screen

[2719] Step 3:

[2720] User: Taps a text field to enter new task information.

[2721] Input: Tap operation

[2722] Output: Focus on text field

[2723] Step 4:

[2724] User: Enter task details in the text field and tap the "Done" button.

[2725] Input: Task information

[2726] Output: Task information input completed

[2727] Step 5:

[2728] Terminal: Sends the entered task details to the server.

[2729] Specifically, it generates an HTTP POST request and sends it to the server's task management API endpoint.

[2730] Input: Completed entering task information

[2731] Output: Task add request

[2732] Step 6:

[2733] Server: Analyzes the received task information and registers it in the database.

[2734] As a specific operation, an INSERT query is executed on the database.

[2735] Input: Task Add Request

[2736] Output: Database update results

[2737] Step 7:

[2738] Server: If registration in the database is successful, it sends a success message to the terminal.

[2739] Specifically, a success status is returned in the HTTP response.

[2740] Input: Database update results

[2741] Output: Success message

[2742] Step 8:

[2743] Terminal: Receives the success message and updates the task management screen to show the new task information.

[2744] Specific operations include updating local data and redrawing the user interface.

[2745] Input: Success message

[2746] Output: Updated task information

[2747] emotion recognition

[2748] Step 1:

[2749] User: Facial expressions and voice are input by the emotion engine during operation.

[2750] Input: Facial expressions and voice data

[2751] Output: Input to the emotion engine

[2752] Step 2:

[2753] Terminal: The emotion engine analyzes the input data and recognizes the user's emotions.

[2754] Specifically, it uses machine learning models to classify emotions.

[2755] Input: Facial expressions and voice data

[2756] Output: Recognized emotion data

[2757] Step 3:

[2758] Terminal: Sends the recognized emotion data to the server.

[2759] Specifically, it creates emotion data in JSON format and generates and sends an HTTP POST request.

[2760] Input: Recognized emotion data

[2761] Output: Sending emotion data

[2762] Step 4:

[2763] Server: Receives emotion data and sends instructions back to the device to dynamically adjust the interface display and operation method.

[2764] Specifically, it generates UI change instructions according to the emotion and returns them as an HTTP response.

[2765] Input: Emotion data

[2766] Output: UI change instructions

[2767] Step 5:

[2768] Terminal: Receives instructions from the server and dynamically changes the interface to adapt to the user's current emotional state.

[2769] Specific actions include changing the layout and color of the interface.

[2770] Input: UI change instructions

[2771] Output: Dynamically changed interface

[2772] (Application example 2)

[2773] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2774] Conventional systems lack an operational interface that takes user emotions into account, which means that operability declines when users feel stressed. Furthermore, there is a lack of systems to improve individual customer experiences in brick-and-mortar stores, and there is also the problem that customization suggestions based on user emotions have not yet been fully implemented.

[2775] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2776] In this invention, the server includes an operation terminal operated by a user, a central processing unit that processes input information received from the operation terminal, an information storage device that stores information stored in the central processing unit, a display unit that displays tasks on the operation terminal, a unit that adjusts schedules and subdivides tasks through selection operations on the display unit, an emotion recognition unit that analyzes the user's facial expression and voice data, and a unit that dynamically changes the content of the display unit based on the emotion identified by the emotion recognition unit. This makes it possible to dynamically adjust the interface according to the user's emotion and improve the individual customer experience in physical stores.

[2777] An "operation terminal" is a device that is directly operated by a user, such as a smartphone or tablet.

[2778] The "central processing unit" is a device that processes input information received from an operation terminal, and is usually a computer system called a server.

[2779] "Information storage device" means a device for storing data saved in a central processing unit, and includes a database.

[2780] The "display means" is an interface for displaying tasks and events on the operating terminal, and is generally a touch screen display.

[2781] The "selection operation" refers to an action in which a user selects a task or event on the display means, and includes finger movements such as tapping and swiping.

[2782] "Emotion recognition means" is a technology that identifies emotions by analyzing a user's facial expressions and voice data, and machine learning models are often used.

[2783] The "dynamically changing means" refers to a function of adjusting the content of the display means in real time based on the emotion identified by the emotion recognition means.

[2784] The embodiment of the present invention is a system comprising an operation terminal operated by a user, a central processing unit, an information storage device, a display means, a selection operation means, an emotion recognition means, and a dynamic change means. A specific embodiment of this system will be described in detail below.

[2785] System Configuration

[2786] 1. Operation terminal: A device that is directly operated by the user, such as a smartphone or tablet. It is equipped with a touch screen, camera, and microphone, and can capture the user's facial expressions and voice.

[2787] 2. Central Processing Unit: A computer system that processes input information received from the operating terminal, usually called a server. It integrates multiple input information and retrieves and registers data from the information storage device.

[2788] 3. Information storage device: A device that includes a database and stores schedule information, task information, user authentication information, emotion data, etc.

[2789] 4. Display: An interface displayed on the operating device, providing a visual display to intuitively show tasks and events. It accepts operations such as tapping and swiping.

[2790] 5. Selection operation means: This function detects tap and release operations performed by the user on the display means and allows for schedule adjustments and task subdivision.

[2791] 6. Emotion recognition: Analyzes facial expressions and voice data acquired from the smartphone camera and microphone to identify the user's emotions. Uses machine learning models (e.g., TensorFlow, PyTorch).

[2792] 7. Dynamic modification: This function adjusts the content of the display means in real time based on the emotional data identified by the emotion recognition means. The interface is modified in a way that adapts to the user's current emotional state.

[2793] Program processing explanation

[2794] 1. User: Uses a smartphone to launch the application and operate the interface shown on the display means.

[2795] 2. Device: Detects selection operations such as taps and swipes performed by the user. Also, captures the user's facial expressions and voice using a camera and microphone. This data is sent to the emotion recognition system.

[2796] 3. Emotion recognition means: Analyzes facial expressions and voice data acquired using a machine learning model to identify the user's emotions. The identified emotion data is sent to the central processing unit.

[2797] 4. Central processing unit: After receiving the emotion data, it retrieves related data from the information storage device and generates content to be sent to the display means. It also generates product suggestions and schedule information according to the emotion.

[2798] 5. Display means: Based on the data sent from the central processing unit, the interface content is dynamically changed and displayed to the user, such as special product offers, event information, or relaxing color themes.

[2799] Specific examples

[2800] For example, if a user stops in front of a particular product shelf in a physical store and looks confused, the smartphone's camera and microphone will capture this. The emotion recognition means will identify the emotion as "confusion" and send that data to the central processing unit. The central processing unit will then generate an interface that provides the confused user with detailed product information and usage instructions, and display this on the display means in real time.

[2801] Prompt Sentence Examples

[2802] "Users' emotions are confusing. Can you recommend any products?"

[2803] This system allows the interface to dynamically adjust based on the user's emotions, significantly improving the personalized customer experience in physical stores.

[2804] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2805] Step 1:

[2806] The user starts the application using a smartphone, enters the username and password on the login screen, and taps the "Login" button.

[2807] Input: Username, Password

[2808] Output: Generated authentication request

[2809] Specifically, it detects input on the smartphone's touchscreen and creates an authentication request.

[2810] Step 2:

[2811] The device sends an authentication request to the server.

[2812] Input: Authentication request

[2813] Output: Send data to the server

[2814] Specifically, the network function of the smartphone is used to send authentication request data to the server.

[2815] Step 3:

[2816] Based on the request received by the server, the information storage device checks the user name and password.

[2817] Input: Authentication request (username, password)

[2818] Output: Authentication result (success or failure)

[2819] Specifically, it performs a database query and compares the received data with the stored data.

[2820] Step 4:

[2821] If the server is successful in the authentication, it generates an authentication token and sends it to the terminal as a response.

[2822] Input: Authentication result (success)

[2823] Output: Authentication token

[2824] Specifically, the authentication token is generated using a random number generation algorithm and sent to the terminal.

[2825] Step 5:

[2826] The device receives the authentication token, stores it locally, and displays the home screen.

[2827] Input: Authentication token

[2828] Output: Home screen display

[2829] Specifically, it stores the token in local storage and updates the application's UI component to display the home screen.

[2830] Step 6:

[2831] As the user walks around the store, the device uses a camera and microphone to capture the user's facial expressions and voice.

[2832] Input: Camera video, audio data

[2833] Output: Captured data

[2834] Specifically, it periodically collects data from the camera and microphone sensors.

[2835] Step 7:

[2836] An emotion recognition means in the terminal analyzes the captured data and identifies the user's emotion.

[2837] Input: Captured data (facial expression, voice)

[2838] Output: Recognized emotion data

[2839] Specifically, it uses machine learning models to analyze data and identify specific emotions.

[2840] Step 8:

[2841] The device transmits the recognized emotion data to the server.

[2842] Input: Emotion data

[2843] Output: Send emotion data to the server

[2844] As a specific operation, emotion data is transmitted via a network.

[2845] Step 9:

[2846] The server receives the emotion data and uses a generative AI model to generate data for appropriate product suggestions and interface changes.

[2847] Input: Emotion data

[2848] Output: Proposal data, interface change data

[2849] Specifically, the prompt sentence "The user is confused. Please recommend some products." is input into the AI ​​model, and the generated suggestions are obtained.

[2850] Step 10:

[2851] The server transmits the generated proposal data to the terminal.

[2852] Input: Proposal data

[2853] Output: Sending data to the terminal

[2854] As a specific operation, the proposal data is transmitted to the terminal via the network.

[2855] Step 11:

[2856] The terminal receives the proposal data and dynamically adjusts and presents an interface to the user using the display means.

[2857] Input: Proposal data

[2858] Output: Dynamically adjusted interface

[2859] Specifically, it updates interface elements such as display content, color themes, and messages and displays them to the user.

[2860] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2861] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2862] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2863] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2864] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2865] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2866] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2867] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, motorcycles, and other devices, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2868] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2869] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2870] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2871] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2872] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2873] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2874] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2875] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2876] The hardware resource ...

Claims

1. an operation terminal operated by a user; a server that processes input information received from the operation terminal; a database stored on the server; an interface that displays events and tasks on the operation terminal; a means for adjusting schedules and detailing tasks by tapping and releasing on the interface; A system including:

2. The server includes a means for verifying authentication information received from the operation terminal, generating an authentication token, and transmitting the generated token to the operation terminal. The system of claim 1 .

3. the operation terminal is provided with means for transmitting the event information and time slot information acquired based on the tap-and-release operation to the server; The system of claim 1 .

Images