System

Abstract

A system is provided.SOLUTION: A system comprising: means for processing a book using a generative model; means for notifying a result generated by the generative model; means for consolidating notifications of a plurality of applications; and means for optimizing user operations.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Today's smartphone users must receive and respond to notifications from multiple applications independently. This not only results in receiving many unnecessary notifications, but also in the risk of overlooking important notifications. Responding to notifications from each application requires multiple operations, which is a burden for users. Furthermore, these operations can be extremely cumbersome, especially for elderly users and those who are not good at digital operations, and are a factor in discouraging the use of digital devices. To solve these issues, a system is needed that centrally manages notifications and operations for each application on a smartphone and streamlines user operations. [Means for solving the problem]

[0005] The present invention provides a means for processing books using a generative model and notifying users of the results generated by the generative model. It also includes a means for unifying notifications from multiple applications and optimizing user operations. This system allows users to check notifications from multiple applications from a single interface and respond efficiently based on summaries and alternative suggestions provided by the generative model. This reduces the burden on users and promotes the use of digital devices. This system is particularly effective in processing email and traffic information, simplifying the information users receive and allowing them to respond promptly as needed.

[0006] A "generative model" is an algorithm that generates, analyzes, and summarizes data based on machine learning or deep learning.

[0007] "Books" refers to a collection of information or data provided in digital form, including, for example, e-mail and traffic information.

[0008] A "notification" is information sent from an application to a user that is presented in a format that is easy for the user to understand.

[0009] The term "means" refers to specific functions, devices, and system components for realizing the present invention.

[0010] "Unified management" means managing and manipulating information from multiple different applications and data sources in a centralized manner on a single interface or system.

[0011] "User" refers to an individual or organization that uses this system and enjoys the functions provided by the system.

[0012] "Optimization" refers to adjusting and improving a system so that users can operate and respond most efficiently and effectively. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The system of the present invention includes a means for processing books using a generative model, a means for notifying the results generated by the generative model, a means for unifying notifications from multiple applications, and a means for optimizing user operations. The program processing flow of this system is explained in natural language, with specific examples.

[0035] System Overview

[0036] This system consists of a server, the user's smartphone (device), and a generative model. Notifications and operations of each application are centrally managed, and the generative model automatically handles the necessary operations.

[0037] Overview of the embodiment

[0038] User authentication and account information acquisition

[0039] When a user logs in to the app, the server calls the authentication API to authenticate the user. If authentication is successful, the server sends a request to the corresponding API to obtain account information for each of the user's applications, encrypts the account information returned from each application, and stores it in the database.

[0040] Retrieving application notifications and sending them to the server

[0041] The device receives notifications from each application installed on the device, temporarily stores the received notifications in a buffer, and encrypts and transmits the notification data in the buffer to a server at regular intervals.

[0042] Analyzing notifications and submitting them to a generated AI model

[0043] The server analyzes the received notification, identifies the notification type, and determines whether processing is necessary. Based on the analyzed notification, the server creates a request to the generative AI model for processing and sends the data to the generative AI model for analysis.

[0044] Processing notification content and generating instructions and replies

[0045] The generative model analyzes the data received from the server and generates appropriate summaries, instructions, and replies. For example, in the case of emails, it summarizes the text, and in the case of traffic information, it generates alternative transportation methods for cancelled services. The generated summaries and instructions are then sent back to the server.

[0046] Sending generated instructions and replies from the server to the device

[0047] The server organizes the summaries and instructions received from the generative model, encrypts them, and sends them to the device.

[0048] What to show the user and how to take action

[0049] The terminal decodes the summary and instructions received from the server and displays them to the user. The user then checks the displayed summary and instructions and decides on the next action. The terminal then receives the user's action and performs the necessary operation. After the operation is completed, it reports the result to the server.

[0050] Specific examples

[0051] For the Mail app

[0052] 1. The server obtains the user's email account information, encrypts it, and stores it.

[0053] 2. Your device will receive a notification of new email.

[0054] 3. The device sends the notification to the server.

[0055] 4. The server parses the email notification and sends the email body to the generative model.

[0056] 5. The generative model summarizes the email body and returns the summary results to the server.

[0057] 6. The server sends the summary results to the terminal.

[0058] 7. The terminal displays the summary to the user.

[0059] 8. The user reviews the summary and selects a reply.

[0060] 9. The device sends the user's reply via the email app.

[0061] 10. The terminal reports the completion of the reply to the server.

[0062] For traffic information apps

[0063] 1. The server obtains the user's traffic information account information, encrypts it, and stores it.

[0064] 2. The device receives a notification that the train is cancelled.

[0065] 3. The device sends the notification to the server.

[0066] 4. The server analyzes the cancellation notice and sends the cancellation information to the generative model.

[0067] 5. The generative model proposes alternative transportation options for the cancelled service and sends the results back to the server.

[0068] 6. The server sends the proposal results to the terminal.

[0069] 7. The device displays the proposed results to the user.

[0070] 8. The user reviews the proposal and selects an alternative.

[0071] 9. The terminal reserves alternative means based on the user's selection.

[0072] 10. The terminal reports the completion of the reservation to the server.

[0073] This system allows users to centrally manage notifications and their processing from various applications, significantly reducing the operational burden.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] When a user logs in to an app, the server calls the authentication API to authenticate the user. If authentication is successful, the server sends a request to the corresponding API to obtain account information for each application of the user.

[0077] Step 2:

[0078] The server encrypts the account information returned by each application and stores it in a database, ensuring that user authentication information is stored securely.

[0079] Step 3:

[0080] The device receives notifications from each application installed on the device, such as new emails or train schedules.

[0081] Step 4:

[0082] The terminal temporarily stores the received notification in a buffer, thereby temporarily storing the notification information.

[0083] Step 5:

[0084] The terminal encrypts the notification data in the buffer at regular intervals and sends it to the server. Encryption ensures that the notification data is transferred safely.

[0085] Step 6:

[0086] The server analyzes the received notifications, identifies the notification type (e.g., email notification, traffic information notification), and determines the need for processing each notification.

[0087] Step 7:

[0088] When a notification is determined to require processing, the server creates a request to the generative model to request processing.

[0089] Step 8:

[0090] The server sends the notification details (e.g., email body, traffic information details) to the generative model.

[0091] Step 9:

[0092] The generative model analyzes the data received from the server and generates appropriate summaries, instructions, and replies. For example, in the case of emails, it summarizes the text, and in the case of traffic information, it generates alternative transportation options for cancelled services.

[0093] Step 10:

[0094] The generative model sends the generated summary and instructions back to the server.

[0095] Step 11:

[0096] The server organizes the summaries and instructions received from the generative model and sends them to the device. The data is encrypted before being transmitted.

[0097] Step 12:

[0098] The terminal decodes the summary and instructions received from the server and displays them to the user in a format that makes it easy for the user to understand the information.

[0099] Step 13:

[0100] The user reviews the displayed summary and instructions and decides on the next action (e.g., reply to email, select alternative transportation).

[0101] Step 14:

[0102] The device receives the user's action and performs the necessary operation, such as sending a reply to an email or booking transportation.

[0103] Step 15:

[0104] After the operation is completed, the terminal reports the result to the server, allowing the server to grasp the overall processing status.

[0105] Example 1

[0106] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0107] In modern society, users need to use multiple applications and receive various notifications from each. However, processing these notifications individually is cumbersome, time-consuming, and labor-intensive. There is also a risk of missing important notifications. Furthermore, optimizing operations across multiple applications is technically difficult, so an efficient means to reduce the burden on users is needed.

[0108] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0109] In this invention, the server includes means for performing user authentication, means for acquiring account information, means for receiving notifications from the terminal, means for analyzing the notifications, means for causing a generative AI model to process the notification content, means for organizing the generated results, means for transmitting the generated results to the terminal, means for receiving user operations, and means for executing processing based on the user operations. This allows the user to efficiently process centrally managed notifications and perform necessary operations quickly and automatically.

[0110] "User authentication" is a method of verifying the legitimacy of a user accessing a system.

[0111] "Account information" refers to the information required for a user to log in to each application, and typically includes a user name and password.

[0112] "Terminal" means a device operated by a user to receive and display notifications.

[0113] "Notifications" are information alerts or messages sent from each application to the terminal.

[0114] A "server" is a central computer system that handles the processing of the entire system.

[0115] A "generative AI model" is an artificial intelligence algorithm that analyzes the data it receives and generates appropriate summaries, instructions, and replies.

[0116] "Analysis of notifications" refers to the process of understanding the content of received notifications and determining the necessary processing.

[0117] "Result cleanup" is the process of formatting and organizing the results received from the generative model into an appropriate format before displaying them to the user.

[0118] "Performing an operation" is a means of carrying out a specific action based on a user's instructions.

[0119] "Centralized management" is a means of managing notifications and information from multiple applications through a unified interface.

[0120] The system of the present invention processes books using a generative AI model and notifies users of the results generated by the generative model. Specifically, it centrally manages notifications from multiple applications and optimizes user operation. This system consists of a server, a user's device such as a smartphone or tablet, and the generative AI model.

[0121] Hardware and software used

[0122] Server: Server for data processing and storage using cloud services (e.g., Amazon Web Services, Microsoft Azure)

[0123] Terminal: A device that functions as a user interface (e.g., a smartphone or tablet running iOS or Android).

[0124] Generative AI models: Natural language processing models (e.g., GPT-3, BERT)

[0125] User authentication and account information acquisition

[0126] The server authenticates the user using an authentication API (e.g. OAuth 2.0) based on the authentication information (email address and password) entered by the user.

[0127] If authentication is successful, the server uses each application's API to obtain account information, encrypts it with AES, and stores it in the database.

[0128] Retrieving application notifications and sending them to the server

[0129] The device receives notifications from each application that arrives to the user (stored in a temporary buffer), encrypts them with AES at regular intervals, and sends them to the server using the HTTPS protocol.

[0130] Analyzing notifications and submitting them to a generated AI model

[0131] The server analyzes the received notification and identifies the notification type (e.g., email or traffic information).

[0132] Once the required processing is determined, the server makes a request to the generative AI model, sending the data along with a prompt such as "Please summarize the contents of this email."

[0133] Processing notification content and generating instructions and replies

[0134] The generative AI model analyzes the data received from the server and performs the specified task (for example, summarizing emails or suggesting alternative transportation options in response to service suspension information).

[0135] The generated results are sent back to the server.

[0136] Sending generated instructions and replies from the server to the device

[0137] The server organizes the results received from the generative AI model and formats them in a format that is easy for the user to view.

[0138] It is then AES encrypted and sent to the terminal via HTTPS protocol.

[0139] What to show the user and how to take action

[0140] The device decodes the data received from the server and displays it to the user.

[0141] The user checks the displayed data and selects the next action.

[0142] Based on the user's instructions, the terminal performs the necessary operations (for example, replying to an email or booking transportation) and reports the results to the server.

[0143] For example, the following prompts may be used:

[0144] Please summarize the contents of the email.

[0145] "Please suggest alternative transportation for this suspension."

[0146] The system allows users to centralize notifications from multiple applications and respond quickly and efficiently with the help of generative AI models.

[0147] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0148] Step 1:

[0149] User authentication and account information acquisition

[0150] The user enters their email address and password to log in to the app.

[0151] The server calls an authentication API (e.g., OAuth 2.0) based on the entered authentication information and verifies the user's validity.

[0152] If authentication is successful, the server uses each application's API to obtain the user's account information.

[0153] Input: User's email address and password.

[0154] Output: An authentication token and the user's account information (e.g., Google Calendar and Gmail information).

[0155] Specific operation: The server encrypts the acquired account information using AES and stores it in the database.

[0156] Step 2:

[0157] Retrieving application notifications and sending them to the server

[0158] The device receives notifications from each installed application.

[0159] Received notifications are temporarily stored in a buffer.

[0160] At regular intervals, the terminal encrypts the notification data in the buffer using AES and sends it to the server using HTTPS.

[0161] Input: New notifications from each application.

[0162] Output: Encrypted notification data.

[0163] Specific operation: When the device receives a notification of a new email, it temporarily stores the content in a buffer and sends it to the server every 10 minutes.

[0164] Step 3:

[0165] Analyzing notifications and submitting them to a generated AI model

[0166] The server analyzes the received notification and identifies the notification type (e.g., email, traffic information).

[0167] Once the required processing is determined, the server makes a request to the generative AI model.

[0168] For example, the server may send the email body along with a prompt saying "Please summarize the contents of this email."

[0169] Input: Encrypted notification data.

[0170] Output: Parsing requests to the generative AI model.

[0171] What it does: The server parses the email notification and sends the email body along with a prompt to the generative AI model.

[0172] Step 4:

[0173] Processing notification content and generating instructions and replies

[0174] The generative AI model analyzes the notification data received from the server.

[0175] Based on the analysis results, the generative AI model generates summaries, instructions, and replies.

[0176] Input: Analysis request and notification data sent by the server.

[0177] Output: Generated summary, instructions and reply.

[0178] Specific operation: The generative AI model extracts important points from the received email text, generates a summary, and sends the summary back to the server.

[0179] Step 5:

[0180] Sending generated instructions and replies from the server to the device

[0181] The server organizes and formats the summaries and instructions received from the generative AI model.

[0182] They are AES encrypted and sent to the terminal using HTTPS.

[0183] Input: The summary or instructions sent from the generative AI model.

[0184] Output: Encrypted summary or instructions.

[0185] Specific operation: The server formats the summary of the email received from the generated AI model, encrypts it with AES, and sends it to the terminal.

[0186] Step 6:

[0187] What to show the user and how to take action

[0188] The terminal decodes the summary and instructions received from the server and displays them to the user.

[0189] The user checks the displayed summary and instructions and selects the next action.

[0190] Based on the user's instructions, the device performs the necessary operations (e.g., replying to emails or booking transportation) and reports the results to the server.

[0191] Input: Encrypted summary or instructions.

[0192] Output: The user's actions and their results.

[0193] Specific operation: The device decodes the summary received from the server and displays it to the user. After the user presses the reply button, the reply is sent via the email app and the completion is notified to the server.

[0194] (Application example 1)

[0195] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0196] In electronic payment services, it has been difficult to efficiently manage information such as payments, bills, and transfers sent from multiple applications and provide optimal information and operational support to users. Furthermore, the sheer volume of notifications means users run the risk of missing important notifications, making it difficult to respond quickly and appropriately. This results in delays in users' responses to important payments and bills, increasing the likelihood of financial trouble.

[0197] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0198] In this invention, the server includes means for processing books using a generative model, means for notifying results generated by the generative model, means for centrally managing notifications from multiple applications, means for optimizing user operations, and means for centrally managing electronic payment notifications and summarizing and organizing notifications of various payments, bills, and transfers. This allows for efficient processing of a large number of notifications related to electronic payments, enabling users to respond quickly and optimally without missing important information.

[0199] A "generative model" is a model that uses machine learning or deep learning to generate, analyze, or predict data.

[0200] "Books" include information such as documents, papers, e-mails, and traffic information.

[0201] A "notification" is a message that conveys information from each application or system to the user.

[0202] "Centralized management" refers to the aggregation of data and notifications from multiple sources and the handling of them in a unified manner.

[0203] "Optimizing user operations" means enabling the user to complete the operations they need to perform easily and quickly.

[0204] "Electronic Payment Notice" means a notice containing payment, billing, transfer, or other information relating to an Electronic Payment Service.

[0205] A summary is a short summary of detailed information that extracts only the most important points.

[0206] "Organizing" means arranging different information and data in a unified format and order.

[0207] "Information provision" refers to conveying necessary information to users in an appropriate format.

[0208] "Operation support" refers to supporting the operations performed by the user and helping them to proceed efficiently.

[0209] This invention is a system that uses a generative model to efficiently manage electronic payment notifications and provide optimal information to users. This system consists of a server, a user terminal, and a generative AI model.

[0210] When a user logs in to the application, the server authenticates the user using the authentication API. If user authentication is successful, the server encrypts and stores the electronic payment account information in a database. This database is used to store the data required for subsequent notification processing.

[0211] The terminal consists of the user's smartphone or smart glasses and receives notifications from each electronic payment application. The notifications are temporarily stored in a buffer on the terminal and encrypted and sent to the server at regular intervals.

[0212] The server analyzes the notification data received from the device and requests processing from the generative AI model. The generative model generates an appropriate summary and instructions from the analyzed notification content and returns the results to the server. The server organizes the summary and instructions received from the generative model and sends them to the user's device.

[0213] The user's device decodes the summary and instructions received from the server and displays them to the user. The user then checks the displayed summary and instructions and selects the next action. Based on the selected action, the device performs the necessary operation and reports the result to the server after the operation is completed.

[0214] Hardware and software used

[0215] Hardware: User's smartphone, smart glasses

[0216] software:

[0217] Authentication API: A service for user authentication

[0218] Electronic Payment API: Acquisition and notification of electronic payment account information

[0219] Generative AI model: A model service that summarizes notifications and generates optimal instructions

[0220] Encryption Library: Uses Python's cryptography library to encrypt data.

[0221] Specific examples

[0222] When a user uses multiple electronic payment services, they frequently receive payment notices, billing notices, transfer notices, etc. from each service. Managing all of these notices individually can be cumbersome, so by using this system, they can be managed centrally.

[0223] For example, if a user receives a payment notification like this:

[0224] Notification: "Your account has been charged $10,000. Your balance is $50,000."

[0225] When this notification arrives on the device, the notification data is sent to the server, and the generative AI model generates a summary such as, "A payment of ¥10,000 has been made. The balance is ¥50,000."

[0226] Example prompts to send to the generative AI model:

[0227] Parse the notification and generate a summary result and appropriate instructions:

[0228] Notification: "Your account has been charged $10,000. Your balance is $50,000."

[0229] By displaying such summary results to the user, the user can quickly check important information without missing anything, and the necessary next steps are presented, optimizing the user's operations, thereby reducing the risk of financial trouble and improving management efficiency.

[0230] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0231] Step 1:

[0232] When a user logs in to an application, the server authenticates the user using the authentication API. When the user enters their login information (username, password), the authentication API sends that information to the server, and if authentication is successful, an access token is issued. The server receives this access token and confirms that user authentication was successful.

[0233] Step 2:

[0234] The server sends a request to the electronic payment API to get electronic payment account information using the access token. This request includes the user ID and the access token. The electronic payment API returns the corresponding account information, and the server stores the account information in a database using an encryption library.

[0235] Step 3:

[0236] The terminal receives notifications from each electronic payment application using the user's smartphone or smart glasses. When a notification arrives, the terminal temporarily stores the notification in a buffer, and then encrypts the notification data in the buffer and sends it to the server at regular intervals. This encryption is performed using the encryption library mentioned above.

[0237] Step 4:

[0238] The server receives and decodes the notification data sent from the device, analyzes the notification, and creates a request to the generative AI model based on its content. Specifically, it extracts the important parts of the notification to generate a prompt sentence and sends it to the generative AI model service.

[0239] Step 5:

[0240] The generative AI model summarizes the notification content based on the received prompt and generates appropriate instructions. For example, if the notification is "Your account has been paid ¥10,000. The balance is ¥50,000," the generative AI model generates a summary of "Your account has been paid ¥10,000. The balance is ¥50,000" and an instruction to "Check your new income."

[0241] Step 6:

[0242] The server receives the summary and instructions returned by the generative AI model, organizes and encrypts them, and sends the organized data to the user's device, ensuring the integrity and security of the processing results.

[0243] Step 7:

[0244] The terminal decodes the summary and instructions sent by the server and displays them to the user, who then reviews the displayed information and selects the next action to take, such as confirming payment details or approving the charge.

[0245] Step 8:

[0246] The terminal performs the corresponding operation based on the user's selection. For example, if payment confirmation is required, the terminal performs a confirmation operation through the electronic payment application. After this, the terminal reports the operation result to the server and updates the database if necessary.

[0247] This completes the entire flow, allowing the user to centrally manage multiple electronic payment notifications and respond quickly and appropriately.

[0248] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0249] The system of the present invention includes a means for processing books using a generative model and notifying the results generated by the generative model, a means for unifying and managing notifications from multiple applications, a means for optimizing user operations, and an emotion engine for recognizing user emotions. The program processing flow of this system is explained in natural language, with specific examples.

[0250] System Overview

[0251] This system consists of a server, the user's smartphone (device), a generative model, and an emotion engine. It centrally manages notifications and operations for each application, and the generative model automatically processes the necessary operations. By incorporating the emotion engine, it is possible to provide optimal responses according to the user's emotions.

[0252] Overview of the embodiment

[0253] User authentication and account information acquisition

[0254] When a user logs in to an app, the server calls the authentication API to authenticate the user. If authentication is successful, the server sends a request to the corresponding API to obtain account information for each application of the user.

[0255] The server encrypts the account information returned from each application and stores it in a database.

[0256] Retrieving application notifications and sending them to the server

[0257] The device receives notifications from each application installed on the device, temporarily stores the received notifications in a buffer, and encrypts and transmits the notification data in the buffer to a server at regular intervals.

[0258] Parsing notifications and submitting generative models

[0259] The server analyzes the received notifications, identifies the notification type (e.g., email notification, traffic information notification), and determines the need for processing for each. Based on the analyzed notification, it creates a request to the Generative Model for processing and sends the data to be analyzed to the Generative Model.

[0260] Processing notification content and generating instructions and replies

[0261] The generative model analyzes the data received from the server and generates appropriate summaries, instructions, and replies. For example, in the case of emails, it summarizes the text, and in the case of traffic information, it generates alternative transportation methods for cancelled services. The generated summaries and instructions are then sent back to the server.

[0262] Emotion recognition and generative modeling results adjustment

[0263] The device uses an emotion engine to recognize the user's emotions from the user's voice, facial expressions, or text. For example, it uses voice recognition or image recognition to detect the user's current emotions (e.g., joy, sadness, anger, surprise, etc.).

[0264] The generative model adjusts its output depending on the perceived user emotion: for example, if the user is stressed, it can adjust the reply to a calmer, more gentle tone.

[0265] Sending generated instructions and replies from the server to the device

[0266] The server organizes the summaries and instructions received from the generative model and emotion engine, encrypts them, and transmits them to the terminal.

[0267] What to show the user and how to take action

[0268] The terminal decodes the summary and instructions received from the server and displays them to the user in a format that makes it easy for the user to understand the information.

[0269] The user reviews the displayed summary and instructions and decides on the next action (e.g., reply to email, select alternative transportation).

[0270] The device receives the user's action and performs the necessary operation, such as sending a reply to an email or booking transportation.

[0271] After the operation is completed, the terminal reports the result to the server, allowing the server to grasp the overall processing status.

[0272] Specific examples

[0273] For the Mail app

[0274] 1. The server obtains the user's email account information, encrypts it, and stores it.

[0275] 2. Your device will receive a notification of new email.

[0276] 3. The device sends the notification to the server.

[0277] 4. The server parses the email notification and sends the email body to the generative model.

[0278] 5. The generative model summarizes the email body and returns the summary results to the server.

[0279] 6. The device recognizes the user's emotions using an emotion engine (e.g., if the user is feeling stressed).

[0280] 7. The generative model adjusts the reply based on the perceived sentiment.

[0281] 8. The server sends the summary results and the adjusted reply to the terminal.

[0282] 9. The terminal displays the summary to the user and prompts them to reply.

[0283] 10. The user selects a reply and the device sends the reply via the email app.

[0284] 11. The terminal reports the completion of the reply to the server.

[0285] For traffic information apps

[0286] 1. The server obtains the user's traffic information account information, encrypts it, and stores it.

[0287] 2. The device receives a notification that the train is cancelled.

[0288] 3. The device sends the notification to the server.

[0289] 4. The server analyzes the cancellation notice and sends the cancellation information to the generative model.

[0290] 5. The generative model proposes alternative transportation options for the cancelled service and sends the results back to the server.

[0291] 6. The device recognizes the user's emotions using an emotion engine (e.g., if the user is anxious).

[0292] 7. The generative model adjusts the alternative transportation suggestions based on the perceived sentiment.

[0293] 8. The server sends the proposal results to the terminal.

[0294] 9. The device displays the proposed results to the user and encourages them to take action.

[0295] 10. The user reviews the suggestions and selects an alternative.

[0296] 11. The terminal reserves the alternative means and reports the completion of the reservation to the server.

[0297] This system not only allows users to centrally manage notifications and their handling from various applications, but also, by combining it with an emotion engine, enables optimal responses that take into consideration the user's emotions. This system reduces the operational burden and can provide more personal and effective support.

[0298] The processing flow will be explained below.

[0299] Step 1:

[0300] When a user logs in to an app, the server calls the authentication API to authenticate the user. If authentication is successful, the server sends a request to the corresponding API to obtain account information for each application of the user.

[0301] Step 2:

[0302] The server encrypts the account information returned by each application and stores it in a database, ensuring that user authentication information is stored securely.

[0303] Step 3:

[0304] The device receives notifications from each application installed on the device, such as new emails or train schedules.

[0305] Step 4:

[0306] The terminal temporarily stores the received notification in a buffer, thereby temporarily storing the notification information.

[0307] Step 5:

[0308] The terminal encrypts the notification data in the buffer at regular intervals and sends it to the server. Encryption ensures that the notification data is transferred safely.

[0309] Step 6:

[0310] The server analyzes the received notifications, identifies the notification type (e.g., email notification, traffic information notification), and determines the need for processing each notification.

[0311] Step 7:

[0312] When a notification is determined to require processing, the server creates a request to the generative model to request processing.

[0313] Step 8:

[0314] The server sends the notification details (e.g., email body, traffic information details) to the generative model.

[0315] Step 9:

[0316] The generative model analyzes the data received from the server and generates appropriate summaries, instructions, and replies. For example, in the case of emails, it summarizes the text, and in the case of traffic information, it generates alternative transportation options for cancelled services.

[0317] Step 10:

[0318] The generative model sends the generated summary and instructions back to the server.

[0319] Step 11:

[0320] The terminal uses an emotion engine to recognize the user's emotion from the user's voice, facial expression, or text, using, for example, voice recognition, image recognition, and text analysis.

[0321] Step 12:

[0322] The emotion engine identifies the user's emotions and feeds the results back to the generative model. For example, if the user is feeling stressed, the generative model will adjust its results based on that information.

[0323] Step 13:

[0324] The generative model receives feedback from the emotion engine and adjusts the generated summaries and replies, allowing it to provide content that takes the user's emotions into consideration.

[0325] Step 14:

[0326] The server organizes summaries and instructions received from the generative model and emotion engine, and sends them to the device. The data is encrypted before transmission.

[0327] Step 15:

[0328] The terminal decodes the summary and instructions received from the server and displays them to the user in a format that makes it easy for the user to understand the information.

[0329] Step 16:

[0330] The user reviews the displayed summary and instructions and decides on the next action (e.g., reply to email, select alternative transportation).

[0331] Step 17:

[0332] The device receives the user's action and performs the necessary operation, such as sending a reply to an email or booking transportation.

[0333] Step 18:

[0334] After the operation is completed, the terminal reports the result to the server, allowing the server to grasp the overall processing status.

[0335] Example 2

[0336] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0337] Conventional notification management systems handle and manage notifications from multiple applications individually, making user operations cumbersome. Furthermore, they handle notifications uniformly without considering the user's feelings, making it difficult to improve the user experience. Furthermore, the analysis results and responses to notification content were sometimes inappropriate, placing a burden on users.

[0338] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0339] In this invention, the server includes a means for analyzing and processing notifications using a generative model, a means for temporarily storing the analyzed notification content and encrypting and transmitting it at regular intervals, and a means for centrally managing notifications from multiple applications. This simplifies user operations and improves the accuracy of notification content. Furthermore, by combining a means for recognizing emotions from the user's voice, facial expression, or text and a means for adjusting the generation results based on the recognized emotions, the burden of work can be reduced and a personalized and appropriate response can be provided to the user.

[0340] A "generative model" is an algorithm that uses artificial intelligence techniques to analyze input data and generate specific responses or text.

[0341] "Notification" refers to information or alerts provided by each application to the user's device.

[0342] "Encryption" is the process of transforming data into a form that cannot be understood by third parties in order to ensure its security.

[0343] "Emotion" refers to a psychological state determined by analyzing the user's voice, facial expression, text, etc.

[0344] "Centralized management" refers to the centralized management of multiple different pieces of information and processes on a single system or platform.

[0345] "Analysis" is the process of breaking down input data or information and interpreting its meaning.

[0346] "Means" are methods or tools used to achieve a particular function or purpose.

[0347] A "notification buffer" is an area or memory for temporarily storing received notifications.

[0348] "Temporary archiving" is the process of storing data for a short period of time for later processing.

[0349] Our system uses generative models to analyze and process notifications, providing a unified mechanism for managing notifications across multiple applications, streamlining user interaction and enabling personalized responses based on emotion recognition.

[0350] System Configuration

[0351] The system consists of the following components:

[0352] server

[0353] User's smartphone (device)

[0354] Generative Model

[0355] Emotion Engine

[0356] Hardware and Software Details

[0357] server

[0358] The server performs tasks such as user authentication, analysis of notification data, sending requests to the generative model, receiving data from the emotion engine, and managing the database.

[0359] Hardware: A server with a powerful CPU, sufficient memory, and storage

[0360] Software: authentication systems, API request handlers, databases (e.g., MySQL), cryptography libraries (e.g., OpenSSL)

[0361] Terminal

[0362] The device refers to the user's smartphone or tablet, which temporarily stores notification data from installed applications and sends it to a server.

[0363] Hardware: Smartphones, tablets

[0364] Software: Applications (e.g., Gmail app, Google Maps app), notification buffer, SSL / TLS communication

[0365] Generative Model

[0366] The generative model analyzes notification data and generates responses, for example, by using natural language processing (NLP) techniques to generate a summary of the notification content and an appropriate reply.

[0367] Software: OpenAI's GPT-4, etc.

[0368] Emotion Engine

[0369] The emotion engine is responsible for recognizing emotions from the user's voice, facial expressions, text, etc., and providing the results to the generative model.

[0370] Software: Google Speech-to-Text, Amazon Transcribe, OpenCV, Google Cloud Vision API

[0371] Data processing and calculation

[0372] User authentication and account information acquisition

[0373] When a user logs in, the server authenticates the user using the authentication API and obtains the account information for each application of the authenticated user. This information is protected by AES encryption and stored in the database.

[0374] Getting notifications

[0375] The device receives notifications from each application on the device, temporarily stores them in a buffer, and then encrypts the notification data in the buffer and sends it to the server at regular intervals.

[0376] Parsing notifications and processing generative models

[0377] The server parses the notification, identifies the notification type, and then sends a request to the generative model, which then analyzes the notification content and generates an appropriate summary or reply message, which is then sent back to the server.

[0378] Emotion recognition and generation adjustment

[0379] The device recognizes emotions from the user's voice, facial expressions, or text and sends the results to the server, which then integrates data from the generative model and emotion engine to generate an optimized response based on the emotion.

[0380] Specific examples

[0381] For the Mail app

[0382] 1. The server obtains the user's email account information, encrypts it, and stores it.

[0383] 2. Your device will receive a notification of new email.

[0384] 3. The device sends the notification to the server.

[0385] 4. The server parses the email notification and sends the email body to the generative model.

[0386] 5. The generative model summarizes the email body and returns the summary results to the server.

[0387] 6. The device recognizes the user's emotions using an emotion engine.

[0388] 7. The generative model adjusts the reply based on the perceived sentiment.

[0389] 8. The server sends the summary results and the adjusted reply to the terminal.

[0390] 9. The terminal displays the summary to the user and prompts them to reply.

[0391] 10. The user selects a reply and the device sends the reply via the email app.

[0392] 11. The terminal reports the completion of the reply to the server.

[0393] For traffic information apps

[0394] 1. The server obtains the user's traffic information account information, encrypts it, and stores it.

[0395] 2. The device receives a notification that the train is cancelled.

[0396] 3. The device sends the notification to the server.

[0397] 4. The server analyzes the cancellation notice and sends the cancellation information to the generative model.

[0398] 5. The generative model proposes alternative transportation options for the cancelled service and sends the results back to the server.

[0399] 6. The device recognizes the user's emotions using an emotion engine.

[0400] 7. The generative model adjusts the alternative transportation suggestions based on the perceived sentiment.

[0401] 8. The server sends the proposal results to the terminal.

[0402] 9. The device displays the proposed results to the user and encourages them to take action.

[0403] 10. The user reviews the suggestions and selects an alternative.

[0404] 11. The terminal reserves the alternative means and reports the completion of the reservation to the server.

[0405] As a result, this system centralizes the notifications a user receives from various applications and uses a generative model and emotion engine to automatically provide appropriate and personalized responses.

[0406] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0407] Step 1: User authentication and account information acquisition

[0408] When a user logs in to an app, the server accesses the authentication API to authenticate the user. The server receives the user ID and password as input. If authentication is successful, an access token is generated (operation: authentication process). The server then sends a request to the corresponding API to obtain account information for each application of the user (operation: API call). The output is the account information for each application.

[0409] Step 2: Save the account information you obtained

[0410] The server protects the account information returned from each application (e.g., user ID, access token, application setting information) with AES encryption and stores it securely in the database (operation: encryption processing and database storage). The input is the account information of each application, and the output is encrypted account information.

[0411] Step 3: Get notifications for your application

[0412] The device receives notifications from each installed application. Specifically, notifications are received through the OS notification center (Action: Receive Notification). The input is the raw notification from the application, and the output is the notification data stored in the buffer.

[0413] Step 4: Temporarily store and send notification data

[0414] The terminal temporarily stores the acquired notification data in a buffer, and at regular intervals (e.g., every minute), it AES-encrypts the notification data in the buffer and sends it to the server (operation: buffering, encryption, communication). The input is the notification data in the buffer, and the output is the encrypted notification data.

[0415] Step 5: Parse the notification

[0416] The server decrypts the notification data received from the device and analyzes it using natural language processing (NLP) technology to identify the notification type (e.g., email notification, traffic information notification) (operation: data analysis and classification). The input is the encrypted notification data, and the output is the classified notification data and the analysis results.

[0417] Step 6: Submitting the Generative Model

[0418] The server creates a processing request to the generative model based on the analyzed notification data and sends the data (action: create request and send). The input is the analyzed notification data and its classification results, and the output is the request data sent to the generative model.

[0419] Step 7: Response generation using the generative model

[0420] The generative model analyzes notification data received from the server and generates appropriate summaries, instructions, and replies (operations: notification analysis and text generation). Specific operations include using NLP algorithms. The input is notification data from the server, and the output is the generated summary or reply.

[0421] Step 8: Sending the Generated Response Back to the Server

[0422] The generative model returns the generated summary and reply text to the server (action: data transmission). The input is the generated summary and reply text, and the output is the response data received on the server side.

[0423] Step 9: Emotion Recognition

[0424] The device uses an emotion engine to recognize the user's emotions from the user's voice, facial expression, or text (calculation: voice recognition, image recognition). It works by acquiring real-time data using a microphone or camera and performing emotion analysis. The input is the user's voice, facial expression, and text, and the output is recognized emotional data.

[0425] Step 10: Adjusting the results

[0426] The generative model adjusts the generated results based on the recognized user emotion (operation: adjust text tone). The operation includes adjusting the generated text (e.g., changing the tone to a gentler tone). The input is the recognized emotion data and the generated results, and the output is the adjusted response data.

[0427] Step 11: Sending the generated instruction / reply

[0428] The server organizes the summaries and instructions received from the generative model and emotion engine, encrypts them, and sends them to the terminal (operations: data organization, encryption, communication). The input is the adjusted response data and emotion data, and the output is the encrypted data sent to the terminal.

[0429] Step 12: Show the user and take action

[0430] The terminal decodes the summary or instructions received from the server and displays them to the user (action: decode and display). The input is the encrypted response data, and the output is the decoded instructions or summary.

[0431] The user checks the displayed summary and instructions and decides on the next action (action: action selection). The input is the decoded instructions and summary, and the output is the user's selected action.

[0432] The terminal receives the user's action and executes the necessary operation (action: sending a reply or making a reservation). The input is the user's selected action, and the output is the execution result.

[0433] Step 13: Reporting the results

[0434] After the operation is completed, the terminal reports the result to the server (operation: communication and log storage). The input is the execution result of the operation, and the output is the result data reported to the server.

[0435] (Application example 2)

[0436] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0437] In recent years, there has been a demand for efficient management and next-generation work environments on production lines within factories. Furthermore, productivity can be affected by the emotions of workers, so it is important to respond in a way that takes their emotions into consideration. Conventional systems have struggled to recognize workers' emotions in real time, analyze production line data, and propose improvements. This has led to issues such as insufficient optimization of production efficiency and reduction of worker burden.

[0438] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for processing books using a generative model, a means for notifying results generated by the generative model, a means for centrally managing notifications from multiple applications, a means for optimizing user operations, a means for recognizing worker emotions using an emotion engine, a means for adjusting generated results based on the worker emotions, a means for collecting data in real time from devices in the factory, and a means for analyzing the collected data and generating improvement proposals. This enables efficient management of the production line and improvement proposals while taking into consideration the emotions of workers.

[0439] A "generative model" is a machine learning algorithm for generating specific output data from input data.

[0440] "Book" means a document or book containing information in digital form.

[0441] "Notification" means the provision of information from the system to the user.

[0442] "Centralized management" refers to the centralized management of multiple information sources and operations through a single interface.

[0443] The "means for optimizing user operations" is a function for making user operations more efficient and comfortable.

[0444] An "emotion engine" is a system that recognizes a user's emotions by analyzing voice, facial expressions, or text.

[0445] "Means for adjusting generated results based on the worker's emotions" refers to a function that changes the system's responses and suggestions depending on the worker's recognized emotions.

[0446] "Factory equipment" refers to hardware and sensors used on production lines and in manufacturing processes.

[0447] "Means of collecting data in real time" refers to the ability to instantly obtain ongoing data.

[0448] "Means for generating improvement proposals" is a function that generates specific proposals for improving productivity from the results of data analysis.

[0449] System Overview

[0450] The system of the present invention combines a generative model and an emotion engine to efficiently and optimally manage production lines in factories. The system consists of a server, a robot (terminal), a generative model, and an emotion engine. Data is collected in real time from each device, and the generative model analyzes it to generate improvement suggestions and provides instructions to workers via the robot. The emotion engine also recognizes the worker's emotions and adjusts the generated results based on them.

[0451] Hardware and Software

[0452] The server is responsible for centralized data management and analysis. The software running on the server includes a data collection API, data processing scripts, and generative models. The server is equipped with a high-performance database system running Linux and a powerful processor.

[0453] The robot (terminal) collects data from sensors in the factory and sends it to a server. It is also equipped with an emotion engine that analyzes the voice and facial expressions of workers through a facial recognition camera and microphone. The robot recognizes the worker's emotions in real time and sends them to the server along with the collected data.

[0454] The generative model is a machine learning algorithm that runs on a server, analyzes collected data, and generates production line optimization and improvement proposals. The generative model is implemented using machine learning frameworks such as Python and TensorFlow.

[0455] The emotion engine is also software that runs on the server and analyzes the voice and facial expression data sent from the robot to recognize the worker's emotions. The emotion engine uses technologies such as Emotion API and Facial Recognition API.

[0456] Program processing overview

[0457] 1. Data collection and transmission

[0458] The robots collect data in real time from sensors in the factory. The collected data is temporarily stored in a buffer and then sent to a server at regular intervals. The data is encrypted and transmitted securely.

[0459] 2. Data Analysis and Generation

[0460] The server analyzes the received data and extracts performance indicators for each production process. The generative model then generates optimization and improvement proposals for the production line based on the analyzed data and sends the results back to the server.

[0461] 3. Emotion recognition and suggestion adjustment

[0462] The robot collects the worker's voice and facial expressions, and the emotion engine analyzes them to recognize the worker's emotions. The emotion engine then sends the results of the recognition to the server, and the generative model adjusts the generated results based on them.

[0463] 4. Instruction Notification and Feedback

[0464] The server sends the generated improvement suggestions and instructions to the robot, which then presents them to the worker. The worker then takes action based on the suggestions. The robot then provides real-time feedback on the progress and results of the work to the server, which manages the entire system.

[0465] Examples of specific examples and prompts

[0466] Example 1: Worker A during a shift

[0467] If the emotion engine recognizes that worker A is tired, it will convey improvement suggestions obtained from the generative model (e.g., adjusting production speed) in more friendly terms.

[0468] Prompt: "What suggestions would be effective when Worker A is tired?"

[0469] Example 2: Troubleshooting a production line

[0470] In the event of a machine failure, the generative model suggests optimal troubleshooting steps and sends reassuring messages to the human worker via an emotion engine.

[0471] Prompt: "If a machine malfunction occurs, what troubleshooting steps and messages would be effective?"

[0472] As described above, the system of the present invention is capable of efficiently managing and improving a production line while taking into consideration the feelings of workers.

[0473] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0474] Step 1:

[0475] The robot (terminal) collects data in real time from each sensor in the factory. The input data is information about each process on the production line (e.g., temperature, pressure, speed, error rate, etc.). The robot temporarily stores this data in a buffer and performs the necessary preprocessing (e.g., format conversion, noise removal) to unify the data format. The preprocessed data is then stored in the buffer as output.

[0476] Step 2:

[0477] At regular intervals, the robot encrypts the temporarily stored data and sends it to the server. The input is the preprocessed data, and the output is the encrypted data. An encryption algorithm (e.g., AES encryption) is used for the processing. After encryption, the data is securely transferred to the server.

[0478] Step 3:

[0479] The server decrypts the received data and stores it in a database for analysis. The input is encrypted data and the output is decrypted data. A database management system (e.g., MySQL) is used to store the data efficiently.

[0480] Step 4:

[0481] The server analyzes the data stored in the database and extracts performance indicators for each production process. The input is the large amount of production data stored in the database, and the output is the performance indicators resulting from the analysis (e.g., lifting speed, error rate analysis results). Statistical analysis tools (e.g., Python's Pandas library) are used for data analysis.

[0482] Step 5:

[0483] The server requests the generative model to process data based on the performance indicators. The input is the analyzed performance indicators, and the output is the improvement proposals generated by the generative model. The generative model uses a machine learning algorithm (e.g., TensorFlow) to generate an optimal production line operation plan and improvement proposals (e.g., production speed adjustments, troubleshooting procedures).

[0484] Step 6:

[0485] The robot collects the worker's voice and facial expression data and sends it to the emotion engine. The input is the worker's real-time voice and facial expression data, and the output is the emotion recognition results (e.g., joy, anger, fatigue level). Emotion recognition uses voice recognition technology (e.g., Google Speech-to-Text) and image analysis technology (e.g., OpenCV).

[0486] Step 7:

[0487] The emotion engine analyzes voice and facial expression data to recognize the worker's emotions. The input is the voice and facial expression data sent from the robot, and the output is the emotion analysis results. In concrete terms, the voice data is converted into acoustic features, and the facial expression data is analyzed using image recognition technology. The recognized emotions are sent to the server.

[0488] Step 8:

[0489] The server adjusts the results of the generative model based on the sentiment analysis results received from the emotion engine. The inputs are the sentiment analysis results and the proposals generated by the generative model, and the output is the optimal proposal adjusted based on the sentiment. The server adjusts the tone and content of the proposal based on the sentiment analysis results and requests the generative model to process it again.

[0490] Step 9:

[0491] The server sends the adjusted improvement proposal to the robot, which then presents it to the worker. The input is the adjusted improvement proposal, and the output is information displayed in a presentation format (e.g., presentation text, voice message). Specifically, the robot presents information to the worker using a monitor or voice output device.

[0492] Step 10:

[0493] The worker takes action based on the proposed improvement suggestions. The input is the proposed suggestions, and the output is the worker's choice and the action to be taken (e.g., adjusting production speed, restarting the machine). The robot monitors the worker's choices and provides support as needed.

[0494] Step 11:

[0495] The robot provides real-time feedback on the worker's progress and work results to the server. The input is the worker's progress data, and the output is feedback information sent to the server. This allows the server to grasp the overall situation and make further improvement suggestions and optimizations.

[0496] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0497] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0498] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0499] [Second embodiment]

[0500] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0501] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0502] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0503] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0504] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0505] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0506] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0507] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0508] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0509] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0510] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0511] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0512] The system of the present invention includes a means for processing books using a generative model, a means for notifying the results generated by the generative model, a means for unifying notifications from multiple applications, and a means for optimizing user operations. The program processing flow of this system is explained in natural language, with specific examples.

[0513] System Overview

[0514] This system consists of a server, the user's smartphone (device), and a generative model. Notifications and operations of each application are centrally managed, and the generative model automatically handles the necessary operations.

[0515] Overview of the embodiment

[0516] User authentication and account information acquisition

[0517] When a user logs in to the app, the server calls the authentication API to authenticate the user. If authentication is successful, the server sends a request to the corresponding API to obtain account information for each of the user's applications, encrypts the account information returned from each application, and stores it in the database.

[0518] Retrieving application notifications and sending them to the server

[0519] The device receives notifications from each application installed on the device, temporarily stores the received notifications in a buffer, and encrypts and transmits the notification data in the buffer to a server at regular intervals.

[0520] Analyzing notifications and submitting them to a generated AI model

[0521] The server analyzes the received notification, identifies the notification type, and determines whether processing is necessary. Based on the analyzed notification, the server creates a request to the generative AI model for processing and sends the data to the generative AI model for analysis.

[0522] Processing notification content and generating instructions and replies

[0523] The generative model analyzes the data received from the server and generates appropriate summaries, instructions, and replies. For example, in the case of emails, it summarizes the text, and in the case of traffic information, it generates alternative transportation methods for cancelled services. The generated summaries and instructions are then sent back to the server.

[0524] Sending generated instructions and replies from the server to the device

[0525] The server organizes the summaries and instructions received from the generative model, encrypts them, and sends them to the device.

[0526] What to show the user and how to take action

[0527] The terminal decodes the summary and instructions received from the server and displays them to the user. The user then checks the displayed summary and instructions and decides on the next action. The terminal then receives the user's action and performs the necessary operation. After the operation is completed, it reports the result to the server.

[0528] Specific examples

[0529] For the Mail app

[0530] 1. The server obtains the user's email account information, encrypts it, and stores it.

[0531] 2. Your device will receive a notification of new email.

[0532] 3. The device sends the notification to the server.

[0533] 4. The server parses the email notification and sends the email body to the generative model.

[0534] 5. The generative model summarizes the email body and returns the summary results to the server.

[0535] 6. The server sends the summary results to the terminal.

[0536] 7. The terminal displays the summary to the user.

[0537] 8. The user reviews the summary and selects a reply.

[0538] 9. The device sends the user's reply via the email app.

[0539] 10. The terminal reports the completion of the reply to the server.

[0540] For traffic information apps

[0541] 1. The server obtains the user's traffic information account information, encrypts it, and stores it.

[0542] 2. The device receives a notification that the train is cancelled.

[0543] 3. The device sends the notification to the server.

[0544] 4. The server analyzes the cancellation notice and sends the cancellation information to the generative model.

[0545] 5. The generative model proposes alternative transportation options for the cancelled service and sends the results back to the server.

[0546] 6. The server sends the proposal results to the terminal.

[0547] 7. The device displays the proposed results to the user.

[0548] 8. The user reviews the proposal and selects an alternative.

[0549] 9. The terminal reserves alternative means based on the user's selection.

[0550] 10. The terminal reports the completion of the reservation to the server.

[0551] This system allows users to centrally manage notifications and their processing from various applications, significantly reducing the operational burden.

[0552] The processing flow will be explained below.

[0553] Step 1:

[0554] When a user logs in to an app, the server calls the authentication API to authenticate the user. If authentication is successful, the server sends a request to the corresponding API to obtain account information for each application of the user.

[0555] Step 2:

[0556] The server encrypts the account information returned by each application and stores it in a database, ensuring that user authentication information is stored securely.

[0557] Step 3:

[0558] The device receives notifications from each application installed on the device, such as new emails or train schedules.

[0559] Step 4:

[0560] The terminal temporarily stores the received notification in a buffer, thereby temporarily storing the notification information.

[0561] Step 5:

[0562] The terminal encrypts the notification data in the buffer at regular intervals and sends it to the server. Encryption ensures that the notification data is transferred safely.

[0563] Step 6:

[0564] The server analyzes the received notifications, identifies the notification type (e.g., email notification, traffic information notification), and determines the need for processing each notification.

[0565] Step 7:

[0566] When a notification is determined to require processing, the server creates a request to the generative model to request processing.

[0567] Step 8:

[0568] The server sends the notification details (e.g., email body, traffic information details) to the generative model.

[0569] Step 9:

[0570] The generative model analyzes the data received from the server and generates appropriate summaries, instructions, and replies. For example, in the case of emails, it summarizes the text, and in the case of traffic information, it generates alternative transportation options for cancelled services.

[0571] Step 10:

[0572] The generative model sends the generated summary and instructions back to the server.

[0573] Step 11:

[0574] The server organizes the summaries and instructions received from the generative model and sends them to the device. The data is encrypted before being transmitted.

[0575] Step 12:

[0576] The terminal decodes the summary and instructions received from the server and displays them to the user in a format that makes it easy for the user to understand the information.

[0577] Step 13:

[0578] The user reviews the displayed summary and instructions and decides on the next action (e.g., reply to email, select alternative transportation).

[0579] Step 14:

[0580] The device receives the user's action and performs the necessary operation, such as sending a reply to an email or booking transportation.

[0581] Step 15:

[0582] After the operation is completed, the terminal reports the result to the server, allowing the server to grasp the overall processing status.

[0583] Example 1

[0584] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0585] In modern society, users need to use multiple applications and receive various notifications from each. However, processing these notifications individually is cumbersome, time-consuming, and labor-intensive. There is also a risk of missing important notifications. Furthermore, optimizing operations across multiple applications is technically difficult, so an efficient means to reduce the burden on users is needed.

[0586] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0587] In this invention, the server includes means for performing user authentication, means for acquiring account information, means for receiving notifications from the terminal, means for analyzing the notifications, means for causing a generative AI model to process the notification content, means for organizing the generated results, means for transmitting the generated results to the terminal, means for receiving user operations, and means for executing processing based on the user operations. This allows the user to efficiently process centrally managed notifications and perform necessary operations quickly and automatically.

[0588] "User authentication" is a method of verifying the legitimacy of a user accessing a system.

[0589] "Account information" refers to the information required for a user to log in to each application, and typically includes a user name and password.

[0590] "Terminal" means a device operated by a user to receive and display notifications.

[0591] "Notifications" are information alerts or messages sent from each application to the terminal.

[0592] A "server" is a central computer system that handles the processing of the entire system.

[0593] A "generative AI model" is an artificial intelligence algorithm that analyzes the data it receives and generates appropriate summaries, instructions, and replies.

[0594] "Analysis of notifications" refers to the process of understanding the content of received notifications and determining the necessary processing.

[0595] "Result cleanup" is the process of formatting and organizing the results received from the generative model into an appropriate format before displaying them to the user.

[0596] "Performing an operation" is a means of carrying out a specific action based on a user's instructions.

[0597] "Centralized management" is a means of managing notifications and information from multiple applications through a unified interface.

[0598] The system of the present invention processes books using a generative AI model and notifies users of the results generated by the generative model. Specifically, it centrally manages notifications from multiple applications and optimizes user operation. This system consists of a server, a user's device such as a smartphone or tablet, and the generative AI model.

[0599] Hardware and software used

[0600] Server: Server for data processing and storage using cloud services (e.g., Amazon Web Services, Microsoft Azure)

[0601] Terminal: A device that functions as a user interface (e.g., a smartphone or tablet running iOS or Android).

[0602] Generative AI models: Natural language processing models (e.g., GPT-3, BERT)

[0603] User authentication and account information acquisition

[0604] The server authenticates the user using an authentication API (e.g. OAuth 2.0) based on the authentication information (email address and password) entered by the user.

[0605] If authentication is successful, the server uses each application's API to obtain account information, encrypts it with AES, and stores it in the database.

[0606] Retrieving application notifications and sending them to the server

[0607] The device receives notifications from each application that arrives to the user (stored in a temporary buffer), encrypts them with AES at regular intervals, and sends them to the server using the HTTPS protocol.

[0608] Analyzing notifications and submitting them to a generated AI model

[0609] The server analyzes the received notification and identifies the notification type (e.g., email or traffic information).

[0610] Once the required processing is determined, the server makes a request to the generative AI model, sending the data along with a prompt such as "Please summarize the contents of this email."

[0611] Processing notification content and generating instructions and replies

[0612] The generative AI model analyzes the data received from the server and performs the specified task (for example, summarizing emails or suggesting alternative transportation options in response to service suspension information).

[0613] The generated results are sent back to the server.

[0614] Sending generated instructions and replies from the server to the device

[0615] The server organizes the results received from the generative AI model and formats them in a format that is easy for the user to view.

[0616] It is then AES encrypted and sent to the terminal via HTTPS protocol.

[0617] What to show the user and how to take action

[0618] The device decodes the data received from the server and displays it to the user.

[0619] The user checks the displayed data and selects the next action.

[0620] Based on the user's instructions, the terminal performs the necessary operations (for example, replying to an email or booking transportation) and reports the results to the server.

[0621] For example, the following prompts may be used:

[0622] Please summarize the contents of the email.

[0623] "Please suggest alternative transportation for this suspension."

[0624] The system allows users to centralize notifications from multiple applications and respond quickly and efficiently with the help of generative AI models.

[0625] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0626] Step 1:

[0627] User authentication and account information acquisition

[0628] The user enters their email address and password to log in to the app.

[0629] The server calls an authentication API (e.g., OAuth 2.0) based on the entered authentication information and verifies the user's validity.

[0630] If authentication is successful, the server uses each application's API to obtain the user's account information.

[0631] Input: User's email address and password.

[0632] Output: An authentication token and the user's account information (e.g., Google Calendar and Gmail information).

[0633] Specific operation: The server encrypts the acquired account information using AES and stores it in the database.

[0634] Step 2:

[0635] Retrieving application notifications and sending them to the server

[0636] The device receives notifications from each installed application.

[0637] Received notifications are temporarily stored in a buffer.

[0638] At regular intervals, the terminal encrypts the notification data in the buffer using AES and sends it to the server using HTTPS.

[0639] Input: New notifications from each application.

[0640] Output: Encrypted notification data.

[0641] Specific operation: When the device receives a notification of a new email, it temporarily stores the content in a buffer and sends it to the server every 10 minutes.

[0642] Step 3:

[0643] Analyzing notifications and submitting them to a generated AI model

[0644] The server analyzes the received notification and identifies the notification type (e.g., email, traffic information).

[0645] Once the required processing is determined, the server makes a request to the generative AI model.

[0646] For example, the server may send the email body along with a prompt saying "Please summarize the contents of this email."

[0647] Input: Encrypted notification data.

[0648] Output: Parsing requests to the generative AI model.

[0649] What it does: The server parses the email notification and sends the email body along with a prompt to the generative AI model.

[0650] Step 4:

[0651] Processing notification content and generating instructions and replies

[0652] The generative AI model analyzes the notification data received from the server.

[0653] Based on the analysis results, the generative AI model generates summaries, instructions, and replies.

[0654] Input: Analysis request and notification data sent by the server.

[0655] Output: Generated summary, instructions and reply.

[0656] Specific operation: The generative AI model extracts important points from the received email text, generates a summary, and sends the summary back to the server.

[0657] Step 5:

[0658] Sending generated instructions and replies from the server to the device

[0659] The server organizes and formats the summaries and instructions received from the generative AI model.

[0660] They are AES encrypted and sent to the terminal using HTTPS.

[0661] Input: The summary or instructions sent from the generative AI model.

[0662] Output: Encrypted summary or instructions.

[0663] Specific operation: The server formats the summary of the email received from the generated AI model, encrypts it with AES, and sends it to the terminal.

[0664] Step 6:

[0665] What to show the user and how to take action

[0666] The terminal decodes the summary and instructions received from the server and displays them to the user.

[0667] The user checks the displayed summary and instructions and selects the next action.

[0668] Based on the user's instructions, the device performs the necessary operations (e.g., replying to emails or booking transportation) and reports the results to the server.

[0669] Input: Encrypted summary or instructions.

[0670] Output: The user's actions and their results.

[0671] Specific operation: The device decodes the summary received from the server and displays it to the user. After the user presses the reply button, the reply is sent via the email app and the completion is notified to the server.

[0672] (Application example 1)

[0673] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0674] In electronic payment services, it has been difficult to efficiently manage information such as payments, bills, and transfers sent from multiple applications and provide optimal information and operational support to users. Furthermore, the sheer volume of notifications means users run the risk of missing important notifications, making it difficult to respond quickly and appropriately. This results in delays in users' responses to important payments and bills, increasing the likelihood of financial trouble.

[0675] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0676] In this invention, the server includes means for processing books using a generative model, means for notifying results generated by the generative model, means for centrally managing notifications from multiple applications, means for optimizing user operations, and means for centrally managing electronic payment notifications and summarizing and organizing notifications of various payments, bills, and transfers. This allows for efficient processing of a large number of notifications related to electronic payments, enabling users to respond quickly and optimally without missing important information.

[0677] A "generative model" is a model that uses machine learning or deep learning to generate, analyze, or predict data.

[0678] "Books" include information such as documents, papers, e-mails, and traffic information.

[0679] A "notification" is a message that conveys information from each application or system to the user.

[0680] "Centralized management" refers to the aggregation of data and notifications from multiple sources and the handling of them in a unified manner.

[0681] "Optimizing user operations" means enabling the user to complete the operations they need to perform easily and quickly.

[0682] "Electronic Payment Notice" means a notice containing payment, billing, transfer, or other information relating to an Electronic Payment Service.

[0683] A summary is a short summary of detailed information that extracts only the most important points.

[0684] "Organizing" means arranging different information and data in a unified format and order.

[0685] "Information provision" refers to conveying necessary information to users in an appropriate format.

[0686] "Operation support" refers to supporting the operations performed by the user and helping them to proceed efficiently.

[0687] This invention is a system that uses a generative model to efficiently manage electronic payment notifications and provide optimal information to users. This system consists of a server, a user terminal, and a generative AI model.

[0688] When a user logs in to the application, the server authenticates the user using the authentication API. If user authentication is successful, the server encrypts and stores the electronic payment account information in a database. This database is used to store the data required for subsequent notification processing.

[0689] The terminal consists of the user's smartphone or smart glasses and receives notifications from each electronic payment application. The notifications are temporarily stored in a buffer on the terminal and encrypted and sent to the server at regular intervals.

[0690] The server analyzes the notification data received from the device and requests processing from the generative AI model. The generative model generates an appropriate summary and instructions from the analyzed notification content and returns the results to the server. The server organizes the summary and instructions received from the generative model and sends them to the user's device.

[0691] The user's device decodes the summary and instructions received from the server and displays them to the user. The user checks the displayed summary and instructions and selects the next action. Based on the selected action, the device performs the necessary operation and reports the result to the server after the operation is completed.

[0692] Hardware and software used

[0693] Hardware: User's smartphone, smart glasses

[0694] software:

[0695] Authentication API: A service for user authentication

[0696] Electronic Payment API: Acquisition and notification of electronic payment account information

[0697] Generative AI model: A model service that summarizes notifications and generates optimal instructions

[0698] Encryption Library: Uses Python's cryptography library to encrypt data.

[0699] Specific examples

[0700] When a user uses multiple electronic payment services, they frequently receive payment notices, billing notices, transfer notices, etc. from each service. Managing all of these notices individually can be cumbersome, so by using this system, they can be managed centrally.

[0701] For example, if a user receives a payment notification like this:

[0702] Notification: "Your account has been charged $10,000. Your balance is $50,000."

[0703] When this notification arrives on the device, the notification data is sent to the server, and the generative AI model generates a summary such as, "A payment of ¥10,000 has been made. The balance is ¥50,000."

[0704] Example prompts to send to the generative AI model:

[0705] Parse the notification and generate a summary result and appropriate instructions:

[0706] Notification: "Your account has been charged $10,000. Your balance is $50,000."

[0707] By displaying such summary results to the user, the user can quickly check important information without missing anything, and the necessary next steps are presented, optimizing the user's operations, thereby reducing the risk of financial trouble and improving management efficiency.

[0708] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0709] Step 1:

[0710] When a user logs in to an application, the server authenticates the user using the authentication API. When the user enters their login information (username, password), the authentication API sends that information to the server, and if authentication is successful, an access token is issued. The server receives this access token and confirms that user authentication was successful.

[0711] Step 2:

[0712] The server sends a request to the electronic payment API to get electronic payment account information using the access token. This request includes the user ID and the access token. The electronic payment API returns the corresponding account information, and the server stores the account information in a database using an encryption library.

[0713] Step 3:

[0714] The terminal receives notifications from each electronic payment application using the user's smartphone or smart glasses. When a notification arrives, the terminal temporarily stores the notification in a buffer, and then encrypts the notification data in the buffer and sends it to the server at regular intervals. This encryption is performed using the encryption library mentioned above.

[0715] Step 4:

[0716] The server receives and decodes the notification data sent from the device, analyzes the notification, and creates a request to the generative AI model based on its content. Specifically, it extracts the important parts of the notification to generate a prompt sentence and sends it to the generative AI model service.

[0717] Step 5:

[0718] The generative AI model summarizes the notification content based on the received prompt and generates appropriate instructions. For example, if the notification is "Your account has been paid ¥10,000. The balance is ¥50,000," the generative AI model generates a summary of "Your account has been paid ¥10,000. The balance is ¥50,000" and an instruction to "Check your new income."

[0719] Step 6:

[0720] The server receives the summary and instructions returned by the generative AI model, organizes and encrypts them, and sends the organized data to the user's device, ensuring the integrity and security of the processing results.

[0721] Step 7:

[0722] The terminal decodes the summary and instructions sent by the server and displays them to the user, who then reviews the displayed information and selects the next action to take, such as confirming payment details or approving the charge.

[0723] Step 8:

[0724] The terminal performs the corresponding operation based on the user's selection. For example, if payment confirmation is required, the terminal performs a confirmation operation through the electronic payment application. After that, it reports the operation result to the server and updates the database if necessary.

[0725] This completes the entire flow, allowing the user to centrally manage multiple electronic payment notifications and respond quickly and appropriately.

[0726] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0727] The system of the present invention includes a means for processing books using a generative model and notifying the results generated by the generative model, a means for unifying and managing notifications from multiple applications, a means for optimizing user operations, and an emotion engine for recognizing user emotions. The program processing flow of this system is explained in natural language, with specific examples.

[0728] System Overview

[0729] This system consists of a server, the user's smartphone (device), a generative model, and an emotion engine. It centrally manages notifications and operations for each application, and the generative model automatically processes the necessary operations. By incorporating the emotion engine, it is possible to provide optimal responses according to the user's emotions.

[0730] Overview of the embodiment

[0731] User authentication and account information acquisition

[0732] When a user logs in to an app, the server calls the authentication API to authenticate the user. If authentication is successful, the server sends a request to the corresponding API to obtain account information for each application of the user.

[0733] The server encrypts the account information returned from each application and stores it in a database.

[0734] Retrieving application notifications and sending them to the server

[0735] The device receives notifications from each application installed on the device, temporarily stores the received notifications in a buffer, and encrypts and transmits the notification data in the buffer to a server at regular intervals.

[0736] Parsing notifications and submitting generative models

[0737] The server analyzes the received notifications, identifies the notification type (e.g., email notification, traffic information notification), and determines the need for processing for each. Based on the analyzed notification, it creates a request to the Generative Model for processing and sends the data to be analyzed to the Generative Model.

[0738] Processing notification content and generating instructions and replies

[0739] The generative model analyzes the data received from the server and generates appropriate summaries, instructions, and replies. For example, in the case of emails, it summarizes the text, and in the case of traffic information, it generates alternative transportation methods for cancelled services. The generated summaries and instructions are then sent back to the server.

[0740] Emotion recognition and generative modeling results adjustment

[0741] The device uses an emotion engine to recognize the user's emotions from the user's voice, facial expressions, or text. For example, it uses voice recognition or image recognition to detect the user's current emotions (e.g., joy, sadness, anger, surprise, etc.).

[0742] The generative model adjusts its output depending on the perceived user emotion: for example, if the user is stressed, it can adjust the reply to a calmer, more gentle tone.

[0743] Sending generated instructions and replies from the server to the device

[0744] The server organizes the summaries and instructions received from the generative model and emotion engine, encrypts them, and transmits them to the terminal.

[0745] What to show the user and how to take action

[0746] The terminal decodes the summary and instructions received from the server and displays them to the user in a format that makes it easy for the user to understand the information.

[0747] The user reviews the displayed summary and instructions and decides on the next action (e.g., reply to email, select alternative transportation).

[0748] The device receives the user's action and performs the necessary operation, such as sending a reply to an email or booking transportation.

[0749] After the operation is completed, the terminal reports the result to the server, allowing the server to grasp the overall processing status.

[0750] Specific examples

[0751] For the Mail app

[0752] 1. The server obtains the user's email account information, encrypts it, and stores it.

[0753] 2. Your device will receive a notification of new email.

[0754] 3. The device sends the notification to the server.

[0755] 4. The server parses the email notification and sends the email body to the generative model.

[0756] 5. The generative model summarizes the email body and returns the summary results to the server.

[0757] 6. The device recognizes the user's emotions using an emotion engine (e.g., if the user is feeling stressed).

[0758] 7. The generative model adjusts the reply based on the perceived sentiment.

[0759] 8. The server sends the summary results and the adjusted reply to the terminal.

[0760] 9. The terminal displays the summary to the user and prompts them to reply.

[0761] 10. The user selects a reply and the device sends the reply via the email app.

[0762] 11. The terminal reports the completion of the reply to the server.

[0763] For traffic information apps

[0764] 1. The server obtains the user's traffic information account information, encrypts it, and stores it.

[0765] 2. The device receives a notification that the train is cancelled.

[0766] 3. The device sends the notification to the server.

[0767] 4. The server analyzes the cancellation notice and sends the cancellation information to the generative model.

[0768] 5. The generative model proposes alternative transportation options for the cancelled service and sends the results back to the server.

[0769] 6. The device recognizes the user's emotions using an emotion engine (e.g., if the user is anxious).

[0770] 7. The generative model adjusts the alternative transportation suggestions based on the perceived sentiment.

[0771] 8. The server sends the proposal results to the terminal.

[0772] 9. The device displays the proposed results to the user and encourages them to take action.

[0773] 10. The user reviews the suggestions and selects an alternative.

[0774] 11. The terminal reserves the alternative means and reports the completion of the reservation to the server.

[0775] This system not only allows users to centrally manage notifications and their handling from various applications, but also, by combining it with an emotion engine, enables optimal responses that take into consideration the user's emotions. This system reduces the operational burden and can provide more personal and effective support.

[0776] The processing flow will be explained below.

[0777] Step 1:

[0778] When a user logs in to an app, the server calls the authentication API to authenticate the user. If authentication is successful, the server sends a request to the corresponding API to obtain account information for each application of the user.

[0779] Step 2:

[0780] The server encrypts the account information returned by each application and stores it in a database, ensuring that user authentication information is stored securely.

[0781] Step 3:

[0782] The device receives notifications from each application installed on the device, such as new emails or train schedules.

[0783] Step 4:

[0784] The terminal temporarily stores the received notification in a buffer, thereby temporarily storing the notification information.

[0785] Step 5:

[0786] The terminal encrypts the notification data in the buffer at regular intervals and sends it to the server. Encryption ensures that the notification data is transferred safely.

[0787] Step 6:

[0788] The server analyzes the received notifications, identifies the notification type (e.g., email notification, traffic information notification), and determines the need for processing each notification.

[0789] Step 7:

[0790] When a notification is determined to require processing, the server creates a request to the generative model to request processing.

[0791] Step 8:

[0792] The server sends the notification details (e.g., email body, traffic information details) to the generative model.

[0793] Step 9:

[0794] The generative model analyzes the data received from the server and generates appropriate summaries, instructions, and replies. For example, in the case of emails, it summarizes the text, and in the case of traffic information, it generates alternative transportation options for cancelled services.

[0795] Step 10:

[0796] The generative model sends the generated summary and instructions back to the server.

[0797] Step 11:

[0798] The terminal uses an emotion engine to recognize the user's emotion from the user's voice, facial expression, or text, using, for example, voice recognition, image recognition, and text analysis.

[0799] Step 12:

[0800] The emotion engine identifies the user's emotions and feeds the results back to the generative model. For example, if the user is feeling stressed, the generative model will adjust its results based on that information.

[0801] Step 13:

[0802] The generative model receives feedback from the emotion engine and adjusts the generated summaries and replies, allowing it to provide content that takes the user's emotions into consideration.

[0803] Step 14:

[0804] The server organizes summaries and instructions received from the generative model and emotion engine, and sends them to the device. The data is encrypted before transmission.

[0805] Step 15:

[0806] The terminal decodes the summary and instructions received from the server and displays them to the user in a format that makes it easy for the user to understand the information.

[0807] Step 16:

[0808] The user reviews the displayed summary and instructions and decides on the next action (e.g., reply to email, select alternative transportation).

[0809] Step 17:

[0810] The device receives the user's action and performs the necessary operation, such as sending a reply to an email or booking transportation.

[0811] Step 18:

[0812] After the operation is completed, the terminal reports the result to the server, allowing the server to grasp the overall processing status.

[0813] Example 2

[0814] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0815] Conventional notification management systems handle and manage notifications from multiple applications individually, making user operations cumbersome. Furthermore, they handle notifications uniformly without considering the user's feelings, making it difficult to improve the user experience. Furthermore, the analysis results and responses to notification content were sometimes inappropriate, placing a burden on users.

[0816] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0817] In this invention, the server includes a means for analyzing and processing notifications using a generative model, a means for temporarily storing the analyzed notification content and encrypting and transmitting it at regular intervals, and a means for centrally managing notifications from multiple applications. This simplifies user operations and improves the accuracy of notification content. Furthermore, by combining a means for recognizing emotions from the user's voice, facial expression, or text and a means for adjusting the generation results based on the recognized emotions, the burden of work can be reduced and a personalized and appropriate response can be provided to the user.

[0818] A "generative model" is an algorithm that uses artificial intelligence techniques to analyze input data and generate specific responses or text.

[0819] "Notification" refers to information or alerts provided by each application to the user's device.

[0820] "Encryption" is the process of transforming data into a form that cannot be understood by third parties in order to ensure its security.

[0821] "Emotion" refers to a psychological state determined by analyzing the user's voice, facial expression, text, etc.

[0822] "Centralized management" refers to the centralized management of multiple different pieces of information and processes on a single system or platform.

[0823] "Analysis" is the process of breaking down input data or information and interpreting its meaning.

[0824] "Means" are methods or tools used to achieve a particular function or purpose.

[0825] A "notification buffer" is an area or memory for temporarily storing received notifications.

[0826] "Temporary archiving" is the process of storing data for a short period of time for later processing.

[0827] Our system uses generative models to analyze and process notifications, providing a unified mechanism for managing notifications across multiple applications, streamlining user interaction and enabling personalized responses based on emotion recognition.

[0828] System Configuration

[0829] The system consists of the following components:

[0830] server

[0831] User's smartphone (device)

[0832] Generative Model

[0833] Emotion Engine

[0834] Hardware and Software Details

[0835] server

[0836] The server performs tasks such as user authentication, analysis of notification data, sending requests to the generative model, receiving data from the emotion engine, and managing the database.

[0837] Hardware: A server with a powerful CPU, sufficient memory, and storage

[0838] Software: authentication systems, API request handlers, databases (e.g., MySQL), cryptography libraries (e.g., OpenSSL)

[0839] Terminal

[0840] The device refers to the user's smartphone or tablet, which temporarily stores notification data from installed applications and sends it to a server.

[0841] Hardware: Smartphones, tablets

[0842] Software: Applications (e.g., Gmail app, Google Maps app), notification buffer, SSL / TLS communication

[0843] Generative Model

[0844] The generative model analyzes notification data and generates responses, for example, by using natural language processing (NLP) techniques to generate a summary of the notification content and an appropriate reply.

[0845] Software: OpenAI's GPT-4, etc.

[0846] Emotion Engine

[0847] The emotion engine is responsible for recognizing emotions from the user's voice, facial expressions, text, etc., and providing the results to the generative model.

[0848] Software: Google Speech-to-Text, Amazon Transcribe, OpenCV, Google Cloud Vision API

[0849] Data processing and calculation

[0850] User authentication and account information acquisition

[0851] When a user logs in, the server authenticates the user using the authentication API and obtains the account information for each application of the authenticated user. This information is protected by AES encryption and stored in the database.

[0852] Getting notifications

[0853] The device receives notifications from each application on the device, temporarily stores them in a buffer, and then encrypts the notification data in the buffer and sends it to the server at regular intervals.

[0854] Parsing notifications and processing generative models

[0855] The server parses the notification, identifies the notification type, and then sends a request to the generative model, which then analyzes the notification content and generates an appropriate summary or reply message, which is then sent back to the server.

[0856] Emotion recognition and generation adjustment

[0857] The device recognizes emotions from the user's voice, facial expressions, or text and sends the results to the server, which then integrates data from the generative model and emotion engine to generate an optimized response based on the emotion.

[0858] Specific examples

[0859] For the Mail app

[0860] 1. The server obtains the user's email account information, encrypts it, and stores it.

[0861] 2. Your device will receive a notification of new email.

[0862] 3. The device sends the notification to the server.

[0863] 4. The server parses the email notification and sends the email body to the generative model.

[0864] 5. The generative model summarizes the email body and returns the summary results to the server.

[0865] 6. The device recognizes the user's emotions using an emotion engine.

[0866] 7. The generative model adjusts the reply based on the perceived sentiment.

[0867] 8. The server sends the summary results and the adjusted reply to the terminal.

[0868] 9. The terminal displays the summary to the user and prompts them to reply.

[0869] 10. The user selects a reply and the device sends the reply via the email app.

[0870] 11. The terminal reports the completion of the reply to the server.

[0871] For traffic information apps

[0872] 1. The server obtains the user's traffic information account information, encrypts it, and stores it.

[0873] 2. The device receives a notification that the train is cancelled.

[0874] 3. The device sends the notification to the server.

[0875] 4. The server analyzes the cancellation notice and sends the cancellation information to the generative model.

[0876] 5. The generative model proposes alternative transportation options for the cancelled service and sends the results back to the server.

[0877] 6. The device recognizes the user's emotions using an emotion engine.

[0878] 7. The generative model adjusts the alternative transportation suggestions based on the perceived sentiment.

[0879] 8. The server sends the proposal results to the terminal.

[0880] 9. The device displays the proposed results to the user and encourages them to take action.

[0881] 10. The user reviews the suggestions and selects an alternative.

[0882] 11. The terminal reserves the alternative means and reports the completion of the reservation to the server.

[0883] As a result, this system centralizes the notifications a user receives from various applications and uses a generative model and emotion engine to automatically provide appropriate and personalized responses.

[0884] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0885] Step 1: User authentication and account information acquisition

[0886] When a user logs in to an app, the server accesses the authentication API to authenticate the user. The server receives the user ID and password as input. If authentication is successful, an access token is generated (operation: authentication process). The server then sends a request to the corresponding API to obtain account information for each application of the user (operation: API call). The output is the account information for each application.

[0887] Step 2: Save the account information you obtained

[0888] The server protects the account information returned from each application (e.g., user ID, access token, application setting information) with AES encryption and stores it securely in the database (operation: encryption processing and database storage). The input is the account information of each application, and the output is encrypted account information.

[0889] Step 3: Get notifications for your application

[0890] The device receives notifications from each installed application. Specifically, notifications are received through the OS notification center (Action: Receive Notification). The input is the raw notification from the application, and the output is the notification data stored in the buffer.

[0891] Step 4: Temporarily store and send notification data

[0892] The terminal temporarily stores the acquired notification data in a buffer, and at regular intervals (e.g., every minute), it AES-encrypts the notification data in the buffer and sends it to the server (operation: buffering, encryption, communication). The input is the notification data in the buffer, and the output is the encrypted notification data.

[0893] Step 5: Parse the notification

[0894] The server decrypts the notification data received from the device and analyzes it using natural language processing (NLP) technology to identify the notification type (e.g., email notification, traffic information notification) (operation: data analysis and classification). The input is the encrypted notification data, and the output is the classified notification data and the analysis results.

[0895] Step 6: Submitting the Generative Model

[0896] The server creates a processing request to the generative model based on the analyzed notification data and sends the data (action: create request and send). The input is the analyzed notification data and its classification results, and the output is the request data sent to the generative model.

[0897] Step 7: Response generation using the generative model

[0898] The generative model analyzes notification data received from the server and generates appropriate summaries, instructions, and replies (operations: notification analysis and text generation). Specific operations include using NLP algorithms. The input is notification data from the server, and the output is the generated summary or reply.

[0899] Step 8: Sending the Generated Response Back to the Server

[0900] The generative model returns the generated summary and reply text to the server (action: data transmission). The input is the generated summary and reply text, and the output is the response data received on the server side.

[0901] Step 9: Emotion Recognition

[0902] The device uses an emotion engine to recognize the user's emotions from the user's voice, facial expression, or text (calculation: voice recognition, image recognition). It works by acquiring real-time data using a microphone or camera and performing emotion analysis. The input is the user's voice, facial expression, and text, and the output is recognized emotional data.

[0903] Step 10: Adjusting the results

[0904] The generative model adjusts the generated results based on the recognized user emotion (operation: adjust text tone). The operation includes adjusting the generated text (e.g., changing the tone to a gentler tone). The input is the recognized emotion data and the generated results, and the output is the adjusted response data.

[0905] Step 11: Sending the generated instruction / reply

[0906] The server organizes the summaries and instructions received from the generative model and emotion engine, encrypts them, and sends them to the terminal (operations: data organization, encryption, communication). The input is the adjusted response data and emotion data, and the output is the encrypted data sent to the terminal.

[0907] Step 12: Show the user and take action

[0908] The terminal decodes the summary or instructions received from the server and displays them to the user (action: decode and display). The input is the encrypted response data, and the output is the decoded instructions or summary.

[0909] The user checks the displayed summary and instructions and decides on the next action (action: action selection). The input is the decoded instructions and summary, and the output is the user's selected action.

[0910] The terminal receives the user's action and executes the necessary operation (action: sending a reply or making a reservation). The input is the user's selected action, and the output is the execution result.

[0911] Step 13: Reporting the results

[0912] After the operation is completed, the terminal reports the result to the server (operation: communication and log storage). The input is the execution result of the operation, and the output is the result data reported to the server.

[0913] (Application example 2)

[0914] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0915] In recent years, there has been a demand for efficient management and next-generation work environments on production lines within factories. Furthermore, productivity can be affected by the emotions of workers, so it is important to respond in a way that takes their emotions into consideration. Conventional systems have struggled to recognize workers' emotions in real time, analyze production line data, and propose improvements. This has led to issues such as insufficient optimization of production efficiency and reduction of worker burden.

[0916] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for processing books using a generative model, a means for notifying results generated by the generative model, a means for centrally managing notifications from multiple applications, a means for optimizing user operations, a means for recognizing worker emotions using an emotion engine, a means for adjusting generated results based on the worker emotions, a means for collecting data in real time from devices in the factory, and a means for analyzing the collected data and generating improvement proposals. This enables efficient management of the production line and improvement proposals while taking into consideration the emotions of workers.

[0917] A "generative model" is a machine learning algorithm for generating specific output data from input data.

[0918] "Book" means a document or book containing information in digital form.

[0919] "Notification" means the provision of information from the system to the user.

[0920] "Centralized management" refers to the centralized management of multiple information sources and operations through a single interface.

[0921] The "means for optimizing user operations" is a function for making user operations more efficient and comfortable.

[0922] An "emotion engine" is a system that recognizes a user's emotions by analyzing voice, facial expressions, or text.

[0923] "Means for adjusting generated results based on the worker's emotions" refers to a function that changes the system's responses and suggestions depending on the worker's recognized emotions.

[0924] "Factory equipment" refers to hardware and sensors used on production lines and in manufacturing processes.

[0925] "Means of collecting data in real time" refers to the ability to instantly obtain ongoing data.

[0926] "Means for generating improvement proposals" is a function that generates specific proposals for improving productivity from the results of data analysis.

[0927] System Overview

[0928] The system of the present invention combines a generative model and an emotion engine to efficiently and optimally manage production lines in factories. The system consists of a server, a robot (terminal), a generative model, and an emotion engine. Data is collected in real time from each device, and the generative model analyzes it to generate improvement suggestions and provides instructions to workers via the robot. The emotion engine also recognizes the worker's emotions and adjusts the generated results based on them.

[0929] Hardware and Software

[0930] The server is responsible for centralized data management and analysis. The software running on the server includes a data collection API, data processing scripts, and generative models. The server is equipped with a high-performance database system running Linux and a powerful processor.

[0931] The robot (terminal) collects data from sensors in the factory and sends it to a server. It is also equipped with an emotion engine that analyzes the voice and facial expressions of workers through a facial recognition camera and microphone. The robot recognizes the worker's emotions in real time and sends them to the server along with the collected data.

[0932] The generative model is a machine learning algorithm that runs on a server, analyzes collected data, and generates production line optimization and improvement proposals. The generative model is implemented using machine learning frameworks such as Python and TensorFlow.

[0933] The emotion engine is also software that runs on the server and analyzes the voice and facial expression data sent from the robot to recognize the worker's emotions. The emotion engine uses technologies such as Emotion API and Facial Recognition API.

[0934] Program processing overview

[0935] 1. Data collection and transmission

[0936] The robots collect data in real time from sensors in the factory. The collected data is temporarily stored in a buffer and then sent to a server at regular intervals. The data is encrypted and transmitted securely.

[0937] 2. Data Analysis and Generation

[0938] The server analyzes the received data and extracts performance indicators for each production process. The generative model then generates optimization and improvement proposals for the production line based on the analyzed data and sends the results back to the server.

[0939] 3. Emotion recognition and suggestion adjustment

[0940] The robot collects the worker's voice and facial expressions, and the emotion engine analyzes them to recognize the worker's emotions. The emotion engine then sends the results of the recognition to the server, and the generative model adjusts the generated results based on them.

[0941] 4. Instruction Notification and Feedback

[0942] The server sends the generated improvement suggestions and instructions to the robot, which then presents them to the worker. The worker then takes action based on the suggestions. The robot then provides real-time feedback on the progress and results of the work to the server, which manages the entire system.

[0943] Examples of specific examples and prompts

[0944] Example 1: Worker A during a shift

[0945] If the emotion engine recognizes that worker A is tired, it will convey improvement suggestions obtained from the generative model (e.g., adjusting production speed) in more friendly terms.

[0946] Prompt: "What suggestions would be effective when Worker A is tired?"

[0947] Example 2: Troubleshooting a production line

[0948] In the event of a machine failure, the generative model suggests optimal troubleshooting steps and sends reassuring messages to the human worker via an emotion engine.

[0949] Prompt: "If a machine malfunction occurs, what troubleshooting steps and messages would be effective?"

[0950] As described above, the system of the present invention is capable of efficiently managing and improving a production line while taking into consideration the feelings of workers.

[0951] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0952] Step 1:

[0953] The robot (terminal) collects data in real time from each sensor in the factory. The input data is information about each process on the production line (e.g., temperature, pressure, speed, error rate, etc.). The robot temporarily stores this data in a buffer and performs the necessary preprocessing (e.g., format conversion, noise removal) to unify the data format. The preprocessed data is then stored in the buffer as output.

[0954] Step 2:

[0955] At regular intervals, the robot encrypts the temporarily stored data and sends it to the server. The input is the preprocessed data, and the output is the encrypted data. An encryption algorithm (e.g., AES encryption) is used for the processing. After encryption, the data is securely transferred to the server.

[0956] Step 3:

[0957] The server decrypts the received data and stores it in a database for analysis. The input is encrypted data and the output is decrypted data. A database management system (e.g., MySQL) is used to store the data efficiently.

[0958] Step 4:

[0959] The server analyzes the data stored in the database and extracts performance indicators for each production process. The input is the large amount of production data stored in the database, and the output is the performance indicators resulting from the analysis (e.g., lifting speed, error rate analysis results). Statistical analysis tools (e.g., Python's Pandas library) are used for data analysis.

[0960] Step 5:

[0961] The server requests the generative model to process data based on the performance indicators. The input is the analyzed performance indicators, and the output is the improvement proposals generated by the generative model. The generative model uses a machine learning algorithm (e.g., TensorFlow) to generate an optimal production line operation plan and improvement proposals (e.g., production speed adjustments, troubleshooting procedures).

[0962] Step 6:

[0963] The robot collects the worker's voice and facial expression data and sends it to the emotion engine. The input is the worker's real-time voice and facial expression data, and the output is the emotion recognition results (e.g., joy, anger, fatigue level). Emotion recognition uses voice recognition technology (e.g., Google Speech-to-Text) and image analysis technology (e.g., OpenCV).

[0964] Step 7:

[0965] The emotion engine analyzes voice and facial expression data to recognize the worker's emotions. The input is the voice and facial expression data sent from the robot, and the output is the emotion analysis results. In concrete terms, the voice data is converted into acoustic features, and the facial expression data is analyzed using image recognition technology. The recognized emotions are sent to the server.

[0966] Step 8:

[0967] The server adjusts the results of the generative model based on the sentiment analysis results received from the emotion engine. The inputs are the sentiment analysis results and the proposals generated by the generative model, and the output is the optimal proposal adjusted based on the sentiment. The server adjusts the tone and content of the proposal based on the sentiment analysis results and requests the generative model to process it again.

[0968] Step 9:

[0969] The server sends the adjusted improvement proposal to the robot, which then presents it to the worker. The input is the adjusted improvement proposal, and the output is information displayed in a presentation format (e.g., presentation text, voice message). Specifically, the robot presents information to the worker using a monitor or voice output device.

[0970] Step 10:

[0971] The worker takes action based on the proposed improvement suggestions. The input is the proposed suggestions, and the output is the worker's choice and the action to be taken (e.g., adjusting production speed, restarting the machine). The robot monitors the worker's choices and provides support as needed.

[0972] Step 11:

[0973] The robot provides real-time feedback on the worker's progress and work results to the server. The input is the worker's progress data, and the output is feedback information sent to the server. This allows the server to grasp the overall situation and make further improvement suggestions and optimizations.

[0974] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0975] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0976] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0977] [Third embodiment]

[0978] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0979] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0980] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0981] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0982] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0983] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0984] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0985] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0986] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0987] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0988] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0989] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0990] The system of the present invention includes a means for processing books using a generative model, a means for notifying the results generated by the generative model, a means for unifying notifications from multiple applications, and a means for optimizing user operations. The program processing flow of this system is explained in natural language, with specific examples.

[0991] System Overview

[0992] This system consists of a server, the user's smartphone (device), and a generative model. Notifications and operations of each application are centrally managed, and the generative model automatically handles the necessary operations.

[0993] Overview of the embodiment

[0994] User authentication and account information acquisition

[0995] When a user logs in to the app, the server calls the authentication API to authenticate the user. If authentication is successful, the server sends a request to the corresponding API to obtain account information for each of the user's applications, encrypts the account information returned from each application, and stores it in the database.

[0996] Retrieving application notifications and sending them to the server

[0997] The device receives notifications from each application installed on the device, temporarily stores the received notifications in a buffer, and encrypts and transmits the notification data in the buffer to a server at regular intervals.

[0998] Analyzing notifications and submitting them to a generated AI model

[0999] The server analyzes the received notification, identifies the notification type, and determines whether processing is necessary. Based on the analyzed notification, the server creates a request to the generative AI model for processing and sends the data to the generative AI model for analysis.

[1000] Processing notification content and generating instructions and replies

[1001] The generative model analyzes the data received from the server and generates appropriate summaries, instructions, and replies. For example, in the case of emails, it summarizes the text, and in the case of traffic information, it generates alternative transportation methods for cancelled services. The generated summaries and instructions are then sent back to the server.

[1002] Sending generated instructions and replies from the server to the device

[1003] The server organizes the summaries and instructions received from the generative model, encrypts them, and sends them to the device.

[1004] What to show the user and how to take action

[1005] The terminal decodes the summary and instructions received from the server and displays them to the user. The user then checks the displayed summary and instructions and decides on the next action. The terminal then receives the user's action and performs the necessary operation. After the operation is completed, it reports the result to the server.

[1006] Specific examples

[1007] For the Mail app

[1008] 1. The server obtains the user's email account information, encrypts it, and stores it.

[1009] 2. Your device will receive a notification of new email.

[1010] 3. The device sends the notification to the server.

[1011] 4. The server parses the email notification and sends the email body to the generative model.

[1012] 5. The generative model summarizes the email body and returns the summary results to the server.

[1013] 6. The server sends the summary results to the terminal.

[1014] 7. The terminal displays the summary to the user.

[1015] 8. The user reviews the summary and selects a reply.

[1016] 9. The device sends the user's reply via the email app.

[1017] 10. The terminal reports the completion of the reply to the server.

[1018] For traffic information apps

[1019] 1. The server obtains the user's traffic information account information, encrypts it, and stores it.

[1020] 2. The device receives a notification that the train is cancelled.

[1021] 3. The device sends the notification to the server.

[1022] 4. The server analyzes the cancellation notice and sends the cancellation information to the generative model.

[1023] 5. The generative model proposes alternative transportation options for the cancelled service and sends the results back to the server.

[1024] 6. The server sends the proposal results to the terminal.

[1025] 7. The device displays the proposed results to the user.

[1026] 8. The user reviews the proposal and selects an alternative.

[1027] 9. The terminal reserves alternative means based on the user's selection.

[1028] 10. The terminal reports the completion of the reservation to the server.

[1029] This system allows users to centrally manage notifications and their processing from various applications, significantly reducing the operational burden.

[1030] The processing flow will be explained below.

[1031] Step 1:

[1032] When a user logs in to an app, the server calls the authentication API to authenticate the user. If authentication is successful, the server sends a request to the corresponding API to obtain account information for each application of the user.

[1033] Step 2:

[1034] The server encrypts the account information returned by each application and stores it in a database, ensuring that user authentication information is stored securely.

[1035] Step 3:

[1036] The device receives notifications from each application installed on the device, such as new emails or train schedules.

[1037] Step 4:

[1038] The terminal temporarily stores the received notification in a buffer, thereby temporarily storing the notification information.

[1039] Step 5:

[1040] The terminal encrypts the notification data in the buffer at regular intervals and sends it to the server. Encryption ensures that the notification data is transferred safely.

[1041] Step 6:

[1042] The server analyzes the received notifications, identifies the notification type (e.g., email notification, traffic information notification), and determines the need for processing each notification.

[1043] Step 7:

[1044] When a notification is determined to require processing, the server creates a request to the generative model to request processing.

[1045] Step 8:

[1046] The server sends the notification details (e.g., email body, traffic information details) to the generative model.

[1047] Step 9:

[1048] The generative model analyzes the data received from the server and generates appropriate summaries, instructions, and replies. For example, in the case of emails, it summarizes the text, and in the case of traffic information, it generates alternative transportation options for cancelled services.

[1049] Step 10:

[1050] The generative model sends the generated summary and instructions back to the server.

[1051] Step 11:

[1052] The server organizes the summaries and instructions received from the generative model and sends them to the device. The data is encrypted before being transmitted.

[1053] Step 12:

[1054] The terminal decodes the summary and instructions received from the server and displays them to the user in a format that makes it easy for the user to understand the information.

[1055] Step 13:

[1056] The user reviews the displayed summary and instructions and decides on the next action (e.g., reply to email, select alternative transportation).

[1057] Step 14:

[1058] The device receives the user's action and performs the necessary operation, such as sending a reply to an email or booking transportation.

[1059] Step 15:

[1060] After the operation is completed, the terminal reports the result to the server, allowing the server to grasp the overall processing status.

[1061] Example 1

[1062] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1063] In modern society, users need to use multiple applications and receive various notifications from each. However, processing these notifications individually is cumbersome, time-consuming, and labor-intensive. There is also a risk of missing important notifications. Furthermore, optimizing operations across multiple applications is technically difficult, so an efficient means to reduce the burden on users is needed.

[1064] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1065] In this invention, the server includes means for performing user authentication, means for acquiring account information, means for receiving notifications from the terminal, means for analyzing the notifications, means for causing a generative AI model to process the notification content, means for organizing the generated results, means for transmitting the generated results to the terminal, means for receiving user operations, and means for executing processing based on the user operations. This allows the user to efficiently process centrally managed notifications and perform necessary operations quickly and automatically.

[1066] "User authentication" is a method of verifying the legitimacy of a user accessing a system.

[1067] "Account information" refers to the information required for a user to log in to each application, and typically includes a user name and password.

[1068] "Terminal" means a device operated by a user to receive and display notifications.

[1069] "Notifications" are information alerts or messages sent from each application to the terminal.

[1070] A "server" is a central computer system that handles the processing of the entire system.

[1071] A "generative AI model" is an artificial intelligence algorithm that analyzes the data it receives and generates appropriate summaries, instructions, and replies.

[1072] "Analysis of notifications" refers to the process of understanding the content of received notifications and determining the necessary processing.

[1073] "Result cleanup" is the process of formatting and organizing the results received from the generative model into an appropriate format before displaying them to the user.

[1074] "Performing an operation" is a means of carrying out a specific action based on a user's instructions.

[1075] "Centralized management" is a means of managing notifications and information from multiple applications through a unified interface.

[1076] The system of the present invention processes books using a generative AI model and notifies users of the results generated by the generative model. Specifically, it centrally manages notifications from multiple applications and optimizes user operation. This system consists of a server, a user's device such as a smartphone or tablet, and the generative AI model.

[1077] Hardware and software used

[1078] Server: Server for data processing and storage using cloud services (e.g., Amazon Web Services, Microsoft Azure)

[1079] Terminal: A device that functions as a user interface (e.g., a smartphone or tablet running iOS or Android).

[1080] Generative AI models: Natural language processing models (e.g., GPT-3, BERT)

[1081] User authentication and account information acquisition

[1082] The server authenticates the user using an authentication API (e.g. OAuth 2.0) based on the authentication information (email address and password) entered by the user.

[1083] If authentication is successful, the server uses each application's API to obtain account information, encrypts it with AES, and stores it in the database.

[1084] Retrieving application notifications and sending them to the server

[1085] The device receives notifications from each application that arrives to the user (stored in a temporary buffer), encrypts them with AES at regular intervals, and sends them to the server using the HTTPS protocol.

[1086] Analyzing notifications and submitting them to a generated AI model

[1087] The server analyzes the received notification and identifies the notification type (e.g., email or traffic information).

[1088] Once the required processing is determined, the server makes a request to the generative AI model, sending the data along with a prompt such as "Please summarize the contents of this email."

[1089] Processing notification content and generating instructions and replies

[1090] The generative AI model analyzes the data received from the server and performs the specified task (for example, summarizing emails or suggesting alternative transportation options in response to service suspension information).

[1091] The generated results are sent back to the server.

[1092] Sending generated instructions and replies from the server to the device

[1093] The server organizes the results received from the generative AI model and formats them in a format that is easy for the user to view.

[1094] It is then AES encrypted and sent to the terminal via HTTPS protocol.

[1095] What to show the user and how to take action

[1096] The device decodes the data received from the server and displays it to the user.

[1097] The user checks the displayed data and selects the next action.

[1098] Based on the user's instructions, the terminal performs the necessary operations (for example, replying to an email or booking transportation) and reports the results to the server.

[1099] For example, the following prompts may be used:

[1100] Please summarize the contents of the email.

[1101] "Please suggest alternative transportation for this suspension."

[1102] The system allows users to centralize notifications from multiple applications and respond quickly and efficiently with the help of generative AI models.

[1103] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1104] Step 1:

[1105] User authentication and account information acquisition

[1106] The user enters their email address and password to log in to the app.

[1107] The server calls an authentication API (e.g., OAuth 2.0) based on the entered authentication information and verifies the user's validity.

[1108] If authentication is successful, the server uses each application's API to obtain the user's account information.

[1109] Input: User's email address and password.

[1110] Output: An authentication token and the user's account information (e.g., Google Calendar and Gmail information).

[1111] Specific operation: The server encrypts the acquired account information using AES and stores it in the database.

[1112] Step 2:

[1113] Retrieving application notifications and sending them to the server

[1114] The device receives notifications from each installed application.

[1115] Received notifications are temporarily stored in a buffer.

[1116] At regular intervals, the terminal encrypts the notification data in the buffer using AES and sends it to the server using HTTPS.

[1117] Input: New notifications from each application.

[1118] Output: Encrypted notification data.

[1119] Specific operation: When the device receives a notification of a new email, it temporarily stores the content in a buffer and sends it to the server every 10 minutes.

[1120] Step 3:

[1121] Analyzing notifications and submitting them to a generated AI model

[1122] The server analyzes the received notification and identifies the notification type (e.g., email, traffic information).

[1123] Once the required processing is determined, the server makes a request to the generative AI model.

[1124] For example, the server may send the email body along with a prompt saying "Please summarize the contents of this email."

[1125] Input: Encrypted notification data.

[1126] Output: Parsing requests to the generative AI model.

[1127] What it does: The server parses the email notification and sends the email body along with a prompt to the generative AI model.

[1128] Step 4:

[1129] Processing notification content and generating instructions and replies

[1130] The generative AI model analyzes the notification data received from the server.

[1131] Based on the analysis results, the generative AI model generates summaries, instructions, and replies.

[1132] Input: Analysis request and notification data sent by the server.

[1133] Output: Generated summary, instructions and reply.

[1134] Specific operation: The generative AI model extracts important points from the received email text, generates a summary, and sends the summary back to the server.

[1135] Step 5:

[1136] Sending generated instructions and replies from the server to the device

[1137] The server organizes and formats the summaries and instructions received from the generative AI model.

[1138] They are AES encrypted and sent to the terminal using HTTPS.

[1139] Input: The summary or instructions sent from the generative AI model.

[1140] Output: Encrypted summary or instructions.

[1141] Specific operation: The server formats the summary of the email received from the generated AI model, encrypts it with AES, and sends it to the terminal.

[1142] Step 6:

[1143] What to show the user and how to take action

[1144] The terminal decodes the summary and instructions received from the server and displays them to the user.

[1145] The user checks the displayed summary and instructions and selects the next action.

[1146] Based on the user's instructions, the device performs the necessary operations (e.g., replying to emails or booking transportation) and reports the results to the server.

[1147] Input: Encrypted summary or instructions.

[1148] Output: The user's actions and their results.

[1149] Specific operation: The device decodes the summary received from the server and displays it to the user. After the user presses the reply button, the reply is sent via the email app and the completion is notified to the server.

[1150] (Application example 1)

[1151] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1152] In electronic payment services, it has been difficult to efficiently manage information such as payments, bills, and transfers sent from multiple applications and provide optimal information and operational support to users. Furthermore, the sheer volume of notifications means users run the risk of missing important notifications, making it difficult to respond quickly and appropriately. This results in delays in users' responses to important payments and bills, increasing the likelihood of financial trouble.

[1153] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1154] In this invention, the server includes means for processing books using a generative model, means for notifying results generated by the generative model, means for centrally managing notifications from multiple applications, means for optimizing user operations, and means for centrally managing electronic payment notifications and summarizing and organizing notifications of various payments, bills, and transfers. This allows for efficient processing of a large number of notifications related to electronic payments, enabling users to respond quickly and optimally without missing important information.

[1155] A "generative model" is a model that uses machine learning or deep learning to generate, analyze, or predict data.

[1156] "Books" include information such as documents, papers, e-mails, and traffic information.

[1157] A "notification" is a message that conveys information from each application or system to the user.

[1158] "Centralized management" refers to the aggregation of data and notifications from multiple sources and the handling of them in a unified manner.

[1159] "Optimizing user operations" means enabling the user to complete the operations they need to perform easily and quickly.

[1160] "Electronic Payment Notice" means a notice containing payment, billing, transfer, or other information relating to an Electronic Payment Service.

[1161] A summary is a short summary of detailed information that extracts only the most important points.

[1162] "Organizing" means arranging different information and data in a unified format and order.

[1163] "Information provision" refers to conveying necessary information to users in an appropriate format.

[1164] "Operation support" refers to supporting the operations performed by the user and helping them to proceed efficiently.

[1165] This invention is a system that uses a generative model to efficiently manage electronic payment notifications and provide optimal information to users. This system consists of a server, a user terminal, and a generative AI model.

[1166] When a user logs in to the application, the server authenticates the user using the authentication API. If user authentication is successful, the server encrypts and stores the electronic payment account information in a database. This database is used to store the data required for subsequent notification processing.

[1167] The terminal consists of the user's smartphone or smart glasses and receives notifications from each electronic payment application. The notifications are temporarily stored in a buffer on the terminal and encrypted and sent to the server at regular intervals.

[1168] The server analyzes the notification data received from the device and requests processing from the generative AI model. The generative model generates an appropriate summary and instructions from the analyzed notification content and returns the results to the server. The server organizes the summary and instructions received from the generative model and sends them to the user's device.

[1169] The user's device decodes the summary and instructions received from the server and displays them to the user. The user checks the displayed summary and instructions and selects the next action. Based on the selected action, the device performs the necessary operation and reports the result to the server after the operation is completed.

[1170] Hardware and software used

[1171] Hardware: User's smartphone, smart glasses

[1172] software:

[1173] Authentication API: A service for user authentication

[1174] Electronic Payment API: Acquisition and notification of electronic payment account information

[1175] Generative AI model: A model service that summarizes notifications and generates optimal instructions

[1176] Encryption Library: Uses Python's cryptography library to encrypt data.

[1177] Specific examples

[1178] When a user uses multiple electronic payment services, they frequently receive payment notices, billing notices, transfer notices, etc. from each service. Managing all of these notices individually can be cumbersome, so by using this system, they can be managed centrally.

[1179] For example, if a user receives a payment notification like this:

[1180] Notification: "Your account has been charged $10,000. Your balance is $50,000."

[1181] When this notification arrives on the device, the notification data is sent to the server, and the generative AI model generates a summary such as, "A payment of ¥10,000 has been made. The balance is ¥50,000."

[1182] Example prompts to send to the generative AI model:

[1183] Parse the notification and generate a summary result and appropriate instructions:

[1184] Notification: "Your account has been charged $10,000. Your balance is $50,000."

[1185] By displaying such summary results to the user, the user can quickly check important information without missing anything, and the necessary next steps are presented, optimizing the user's operations, thereby reducing the risk of financial trouble and improving management efficiency.

[1186] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1187] Step 1:

[1188] When a user logs in to an application, the server authenticates the user using the authentication API. When the user enters their login information (username, password), the authentication API sends that information to the server, and if authentication is successful, an access token is issued. The server receives this access token and confirms that user authentication was successful.

[1189] Step 2:

[1190] The server sends a request to the electronic payment API to get electronic payment account information using the access token. This request includes the user ID and the access token. The electronic payment API returns the corresponding account information, and the server stores the account information in a database using an encryption library.

[1191] Step 3:

[1192] The terminal receives notifications from each electronic payment application using the user's smartphone or smart glasses. When a notification arrives, the terminal temporarily stores the notification in a buffer, and then encrypts the notification data in the buffer and sends it to the server at regular intervals. This encryption is performed using the encryption library mentioned above.

[1193] Step 4:

[1194] The server receives and decodes the notification data sent from the device, analyzes the notification, and creates a request to the generative AI model based on its content. Specifically, it extracts the important parts of the notification to generate a prompt sentence and sends it to the generative AI model service.

[1195] Step 5:

[1196] The generative AI model summarizes the notification content based on the received prompt and generates appropriate instructions. For example, if the notification is "Your account has been paid ¥10,000. The balance is ¥50,000," the generative AI model generates a summary of "Your account has been paid ¥10,000. The balance is ¥50,000" and an instruction to "Check your new income."

[1197] Step 6:

[1198] The server receives the summary and instructions returned by the generative AI model, organizes and encrypts them, and sends the organized data to the user's device, ensuring the integrity and security of the processing results.

[1199] Step 7:

[1200] The terminal decodes the summary and instructions sent by the server and displays them to the user, who then reviews the displayed information and selects the next action to take, such as confirming payment details or approving the charge.

[1201] Step 8:

[1202] The terminal performs the corresponding operation based on the user's selection. For example, if payment confirmation is required, the terminal performs a confirmation operation through the electronic payment application. After this, the terminal reports the operation result to the server and updates the database if necessary.

[1203] This completes the entire flow, allowing the user to centrally manage multiple electronic payment notifications and respond quickly and appropriately.

[1204] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1205] The system of the present invention includes a means for processing books using a generative model and notifying the results generated by the generative model, a means for unifying and managing notifications from multiple applications, a means for optimizing user operations, and an emotion engine for recognizing user emotions. The program processing flow of this system is explained in natural language, with specific examples.

[1206] System Overview

[1207] This system consists of a server, the user's smartphone (device), a generative model, and an emotion engine. It centrally manages notifications and operations for each application, and the generative model automatically processes the necessary operations. By incorporating the emotion engine, it is possible to provide optimal responses according to the user's emotions.

[1208] Overview of the embodiment

[1209] User authentication and account information acquisition

[1210] When a user logs in to an app, the server calls the authentication API to authenticate the user. If authentication is successful, the server sends a request to the corresponding API to obtain account information for each application of the user.

[1211] The server encrypts the account information returned from each application and stores it in a database.

[1212] Retrieving application notifications and sending them to the server

[1213] The device receives notifications from each application installed on the device, temporarily stores the received notifications in a buffer, and encrypts and transmits the notification data in the buffer to a server at regular intervals.

[1214] Parsing notifications and submitting generative models

[1215] The server analyzes the received notifications, identifies the notification type (e.g., email notification, traffic information notification), and determines the need for processing for each. Based on the analyzed notification, it creates a request to the Generative Model for processing and sends the data to be analyzed to the Generative Model.

[1216] Processing notification content and generating instructions and replies

[1217] The generative model analyzes the data received from the server and generates appropriate summaries, instructions, and replies. For example, in the case of emails, it summarizes the text, and in the case of traffic information, it generates alternative transportation methods for cancelled services. The generated summaries and instructions are then sent back to the server.

[1218] Emotion recognition and generative modeling results adjustment

[1219] The device uses an emotion engine to recognize the user's emotions from the user's voice, facial expressions, or text. For example, it uses voice recognition or image recognition to detect the user's current emotions (e.g., joy, sadness, anger, surprise, etc.).

[1220] The generative model adjusts its output depending on the perceived user emotion: for example, if the user is stressed, it can adjust the reply to a calmer, more gentle tone.

[1221] Sending generated instructions and replies from the server to the device

[1222] The server organizes the summaries and instructions received from the generative model and emotion engine, encrypts them, and transmits them to the terminal.

[1223] What to show the user and how to take action

[1224] The terminal decodes the summary and instructions received from the server and displays them to the user in a format that makes it easy for the user to understand the information.

[1225] The user reviews the displayed summary and instructions and decides on the next action (e.g., reply to email, select alternative transportation).

[1226] The device receives the user's action and performs the necessary operation, such as sending a reply to an email or booking transportation.

[1227] After the operation is completed, the terminal reports the result to the server, allowing the server to grasp the overall processing status.

[1228] Specific examples

[1229] For the Mail app

[1230] 1. The server obtains the user's email account information, encrypts it, and stores it.

[1231] 2. Your device will receive a notification of new email.

[1232] 3. The device sends the notification to the server.

[1233] 4. The server parses the email notification and sends the email body to the generative model.

[1234] 5. The generative model summarizes the email body and returns the summary results to the server.

[1235] 6. The device recognizes the user's emotions using an emotion engine (e.g., if the user is feeling stressed).

[1236] 7. The generative model adjusts the reply based on the perceived sentiment.

[1237] 8. The server sends the summary results and the adjusted reply to the terminal.

[1238] 9. The terminal displays the summary to the user and prompts them to reply.

[1239] 10. The user selects a reply and the device sends the reply via the email app.

[1240] 11. The terminal reports the completion of the reply to the server.

[1241] For traffic information apps

[1242] 1. The server obtains the user's traffic information account information, encrypts it, and stores it.

[1243] 2. The device receives a notification that the train is cancelled.

[1244] 3. The device sends the notification to the server.

[1245] 4. The server analyzes the cancellation notice and sends the cancellation information to the generative model.

[1246] 5. The generative model proposes alternative transportation options for the cancelled service and sends the results back to the server.

[1247] 6. The device recognizes the user's emotions using an emotion engine (e.g., if the user is anxious).

[1248] 7. The generative model adjusts the alternative transportation suggestions based on the perceived sentiment.

[1249] 8. The server sends the proposal results to the terminal.

[1250] 9. The device displays the proposed results to the user and encourages them to take action.

[1251] 10. The user reviews the suggestions and selects an alternative.

[1252] 11. The terminal reserves the alternative means and reports the completion of the reservation to the server.

[1253] This system not only allows users to centrally manage notifications and their handling from various applications, but also, by combining it with an emotion engine, enables optimal responses that take into consideration the user's emotions. This system reduces the operational burden and can provide more personal and effective support.

[1254] The processing flow will be explained below.

[1255] Step 1:

[1256] When a user logs in to an app, the server calls the authentication API to authenticate the user. If authentication is successful, the server sends a request to the corresponding API to obtain account information for each application of the user.

[1257] Step 2:

[1258] The server encrypts the account information returned by each application and stores it in a database, ensuring that user authentication information is stored securely.

[1259] Step 3:

[1260] The device receives notifications from each application installed on the device, such as new emails or train schedules.

[1261] Step 4:

[1262] The terminal temporarily stores the received notification in a buffer, thereby temporarily storing the notification information.

[1263] Step 5:

[1264] The terminal encrypts the notification data in the buffer at regular intervals and sends it to the server. Encryption ensures that the notification data is transferred safely.

[1265] Step 6:

[1266] The server analyzes the received notifications, identifies the notification type (e.g., email notification, traffic information notification), and determines the need for processing each notification.

[1267] Step 7:

[1268] When a notification is determined to require processing, the server creates a request to the generative model to request processing.

[1269] Step 8:

[1270] The server sends the notification details (e.g., email body, traffic information details) to the generative model.

[1271] Step 9:

[1272] The generative model analyzes the data received from the server and generates appropriate summaries, instructions, and replies. For example, in the case of emails, it summarizes the text, and in the case of traffic information, it generates alternative transportation options for cancelled services.

[1273] Step 10:

[1274] The generative model sends the generated summary and instructions back to the server.

[1275] Step 11:

[1276] The terminal uses an emotion engine to recognize the user's emotion from the user's voice, facial expression, or text, using, for example, voice recognition, image recognition, and text analysis.

[1277] Step 12:

[1278] The emotion engine identifies the user's emotions and feeds the results back to the generative model. For example, if the user is feeling stressed, the generative model will adjust its results based on that information.

[1279] Step 13:

[1280] The generative model receives feedback from the emotion engine and adjusts the generated summaries and replies, allowing it to provide content that takes the user's emotions into consideration.

[1281] Step 14:

[1282] The server organizes summaries and instructions received from the generative model and emotion engine, and sends them to the device. The data is encrypted before transmission.

[1283] Step 15:

[1284] The terminal decodes the summary and instructions received from the server and displays them to the user in a format that makes it easy for the user to understand the information.

[1285] Step 16:

[1286] The user reviews the displayed summary and instructions and decides on the next action (e.g., reply to email, select alternative transportation).

[1287] Step 17:

[1288] The device receives the user's action and performs the necessary operation, such as sending a reply to an email or booking transportation.

[1289] Step 18:

[1290] After the operation is completed, the terminal reports the result to the server, allowing the server to grasp the overall processing status.

[1291] Example 2

[1292] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1293] Conventional notification management systems handle and manage notifications from multiple applications individually, making user operations cumbersome. Furthermore, they handle notifications uniformly without considering the user's feelings, making it difficult to improve the user experience. Furthermore, the analysis results and responses to notification content were sometimes inappropriate, placing a burden on users.

[1294] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1295] In this invention, the server includes a means for analyzing and processing notifications using a generative model, a means for temporarily storing the analyzed notification content and encrypting and transmitting it at regular intervals, and a means for centrally managing notifications from multiple applications. This simplifies user operations and improves the accuracy of notification content. Furthermore, by combining a means for recognizing emotions from the user's voice, facial expression, or text and a means for adjusting the generation results based on the recognized emotions, the burden of work can be reduced and a personalized and appropriate response can be provided to the user.

[1296] A "generative model" is an algorithm that uses artificial intelligence techniques to analyze input data and generate specific responses or text.

[1297] "Notification" refers to information or alerts provided by each application to the user's device.

[1298] "Encryption" is the process of transforming data into a form that cannot be understood by third parties in order to ensure its security.

[1299] "Emotion" refers to a psychological state determined by analyzing the user's voice, facial expression, text, etc.

[1300] "Centralized management" refers to the centralized management of multiple different pieces of information and processes on a single system or platform.

[1301] "Analysis" is the process of breaking down input data or information and interpreting its meaning.

[1302] "Means" are methods or tools used to achieve a particular function or purpose.

[1303] A "notification buffer" is an area or memory for temporarily storing received notifications.

[1304] "Temporary archiving" is the process of storing data for a short period of time for later processing.

[1305] Our system uses generative models to analyze and process notifications, providing a unified mechanism for managing notifications across multiple applications, streamlining user interaction and enabling personalized responses based on emotion recognition.

[1306] System Configuration

[1307] The system consists of the following components:

[1308] server

[1309] User's smartphone (device)

[1310] Generative Model

[1311] Emotion Engine

[1312] Hardware and Software Details

[1313] server

[1314] The server performs tasks such as user authentication, analysis of notification data, sending requests to the generative model, receiving data from the emotion engine, and managing the database.

[1315] Hardware: A server with a powerful CPU, sufficient memory, and storage

[1316] Software: authentication systems, API request handlers, databases (e.g., MySQL), cryptography libraries (e.g., OpenSSL)

[1317] Terminal

[1318] The device refers to the user's smartphone or tablet, which temporarily stores notification data from installed applications and sends it to a server.

[1319] Hardware: Smartphones, tablets

[1320] Software: Applications (e.g., Gmail app, Google Maps app), notification buffer, SSL / TLS communication

[1321] Generative Model

[1322] The generative model analyzes notification data and generates responses, for example, by using natural language processing (NLP) techniques to generate a summary of the notification content and an appropriate reply.

[1323] Software: OpenAI's GPT-4, etc.

[1324] Emotion Engine

[1325] The emotion engine is responsible for recognizing emotions from the user's voice, facial expressions, text, etc., and providing the results to the generative model.

[1326] Software: Google Speech-to-Text, Amazon Transcribe, OpenCV, Google Cloud Vision API

[1327] Data processing and calculation

[1328] User authentication and account information acquisition

[1329] When a user logs in, the server authenticates the user using the authentication API and obtains the account information for each application of the authenticated user. This information is protected by AES encryption and stored in the database.

[1330] Getting notifications

[1331] The device receives notifications from each application on the device, temporarily stores them in a buffer, and then encrypts the notification data in the buffer and sends it to the server at regular intervals.

[1332] Parsing notifications and processing generative models

[1333] The server parses the notification, identifies the notification type, and then sends a request to the generative model, which then analyzes the notification content and generates an appropriate summary or reply message, which is then sent back to the server.

[1334] Emotion recognition and generation adjustment

[1335] The device recognizes emotions from the user's voice, facial expressions, or text and sends the results to the server, which then integrates data from the generative model and emotion engine to generate an optimized response based on the emotion.

[1336] Specific examples

[1337] For the Mail app

[1338] 1. The server obtains the user's email account information, encrypts it, and stores it.

[1339] 2. Your device will receive a notification of new email.

[1340] 3. The device sends the notification to the server.

[1341] 4. The server parses the email notification and sends the email body to the generative model.

[1342] 5. The generative model summarizes the email body and returns the summary results to the server.

[1343] 6. The device recognizes the user's emotions using an emotion engine.

[1344] 7. The generative model adjusts the reply based on the perceived sentiment.

[1345] 8. The server sends the summary results and the adjusted reply to the terminal.

[1346] 9. The terminal displays the summary to the user and prompts them to reply.

[1347] 10. The user selects a reply and the device sends the reply via the email app.

[1348] 11. The terminal reports the completion of the reply to the server.

[1349] For traffic information apps

[1350] 1. The server obtains the user's traffic information account information, encrypts it, and stores it.

[1351] 2. The device receives a notification that the train is cancelled.

[1352] 3. The device sends the notification to the server.

[1353] 4. The server analyzes the cancellation notice and sends the cancellation information to the generative model.

[1354] 5. The generative model proposes alternative transportation options for the cancelled service and sends the results back to the server.

[1355] 6. The device recognizes the user's emotions using an emotion engine.

[1356] 7. The generative model adjusts the alternative transportation suggestions based on the perceived sentiment.

[1357] 8. The server sends the proposal results to the terminal.

[1358] 9. The device displays the proposed results to the user and encourages them to take action.

[1359] 10. The user reviews the suggestions and selects an alternative.

[1360] 11. The terminal reserves the alternative means and reports the completion of the reservation to the server.

[1361] As a result, this system centralizes the notifications a user receives from various applications and uses a generative model and emotion engine to automatically provide appropriate and personalized responses.

[1362] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1363] Step 1: User authentication and account information acquisition

[1364] When a user logs in to an app, the server accesses the authentication API to authenticate the user. The server receives the user ID and password as input. If authentication is successful, an access token is generated (operation: authentication process). The server then sends a request to the corresponding API to obtain account information for each application of the user (operation: API call). The output is the account information for each application.

[1365] Step 2: Save the account information you obtained

[1366] The server protects the account information returned from each application (e.g., user ID, access token, application setting information) with AES encryption and stores it securely in the database (operation: encryption processing and database storage). The input is the account information of each application, and the output is encrypted account information.

[1367] Step 3: Get notifications for your application

[1368] The device receives notifications from each installed application. Specifically, notifications are received through the OS notification center (Action: Receive Notification). The input is the raw notification from the application, and the output is the notification data stored in the buffer.

[1369] Step 4: Temporarily store and send notification data

[1370] The terminal temporarily stores the acquired notification data in a buffer, and at regular intervals (e.g., every minute), it AES-encrypts the notification data in the buffer and sends it to the server (operation: buffering, encryption, communication). The input is the notification data in the buffer, and the output is the encrypted notification data.

[1371] Step 5: Parse the notification

[1372] The server decrypts the notification data received from the device and analyzes it using natural language processing (NLP) technology to identify the notification type (e.g., email notification, traffic information notification) (operation: data analysis and classification). The input is the encrypted notification data, and the output is the classified notification data and the analysis results.

[1373] Step 6: Submitting the Generative Model

[1374] The server creates a processing request to the generative model based on the analyzed notification data and sends the data (action: create request and send). The input is the analyzed notification data and its classification results, and the output is the request data sent to the generative model.

[1375] Step 7: Response generation using the generative model

[1376] The generative model analyzes notification data received from the server and generates appropriate summaries, instructions, and replies (operations: notification analysis and text generation). Specific operations include using NLP algorithms. The input is notification data from the server, and the output is the generated summary or reply.

[1377] Step 8: Sending the Generated Response Back to the Server

[1378] The generative model returns the generated summary and reply text to the server (action: data transmission). The input is the generated summary and reply text, and the output is the response data received on the server side.

[1379] Step 9: Emotion Recognition

[1380] The device uses an emotion engine to recognize the user's emotions from the user's voice, facial expression, or text (calculation: voice recognition, image recognition). It works by acquiring real-time data using a microphone or camera and performing emotion analysis. The input is the user's voice, facial expression, and text, and the output is recognized emotional data.

[1381] Step 10: Adjusting the results

[1382] The generative model adjusts the generated results based on the recognized user emotion (operation: adjust text tone). The operation includes adjusting the generated text (e.g., changing the tone to a gentler tone). The input is the recognized emotion data and the generated results, and the output is the adjusted response data.

[1383] Step 11: Sending generated instructions and replies

[1384] The server organizes the summaries and instructions received from the generative model and emotion engine, encrypts them, and sends them to the terminal (operations: data organization, encryption, communication). The input is the adjusted response data and emotion data, and the output is the encrypted data sent to the terminal.

[1385] Step 12: Show the user and take action

[1386] The terminal decodes the summary or instructions received from the server and displays them to the user (action: decode and display). The input is the encrypted response data, and the output is the decoded instructions or summary.

[1387] The user checks the displayed summary and instructions and decides on the next action (action: action selection). The input is the decoded instructions and summary, and the output is the user's selected action.

[1388] The terminal receives the user's action and executes the necessary operation (action: sending a reply or making a reservation). The input is the user's selected action, and the output is the execution result.

[1389] Step 13: Reporting the results

[1390] After the operation is completed, the terminal reports the result to the server (operation: communication and log storage). The input is the execution result of the operation, and the output is the result data reported to the server.

[1391] (Application example 2)

[1392] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1393] In recent years, there has been a demand for efficient management and next-generation work environments on production lines within factories. Furthermore, productivity can be affected by the emotions of workers, so it is important to respond in a way that takes their emotions into consideration. Conventional systems have struggled to recognize workers' emotions in real time, analyze production line data, and propose improvements. This has led to issues such as insufficient optimization of production efficiency and reduction of worker burden.

[1394] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for processing books using a generative model, a means for notifying results generated by the generative model, a means for centrally managing notifications from multiple applications, a means for optimizing user operations, a means for recognizing worker emotions using an emotion engine, a means for adjusting generated results based on the worker emotions, a means for collecting data in real time from devices in the factory, and a means for analyzing the collected data and generating improvement proposals. This enables efficient management of the production line and improvement proposals while taking into consideration the emotions of workers.

[1395] A "generative model" is a machine learning algorithm for generating specific output data from input data.

[1396] "Book" means a document or book containing information in digital form.

[1397] "Notification" means the provision of information from the system to the user.

[1398] "Centralized management" refers to the centralized management of multiple information sources and operations through a single interface.

[1399] The "means for optimizing user operations" is a function for making user operations more efficient and comfortable.

[1400] An "emotion engine" is a system that recognizes a user's emotions by analyzing voice, facial expressions, or text.

[1401] "Means for adjusting generated results based on the worker's emotions" refers to a function that changes the system's responses and suggestions depending on the worker's recognized emotions.

[1402] "Factory equipment" refers to hardware and sensors used on production lines and in manufacturing processes.

[1403] "Means of collecting data in real time" refers to the ability to instantly obtain ongoing data.

[1404] "Means for generating improvement proposals" is a function that generates specific proposals for improving productivity from the results of data analysis.

[1405] System Overview

[1406] The system of the present invention combines a generative model and an emotion engine to efficiently and optimally manage production lines in factories. The system consists of a server, a robot (terminal), a generative model, and an emotion engine. Data is collected in real time from each device, and the generative model analyzes it to generate improvement suggestions and provides instructions to workers via the robot. The emotion engine also recognizes the worker's emotions and adjusts the generated results based on them.

[1407] Hardware and Software

[1408] The server is responsible for centralized data management and analysis. The software running on the server includes a data collection API, data processing scripts, and generative models. The server is equipped with a high-performance database system running Linux and a powerful processor.

[1409] The robot (terminal) collects data from sensors in the factory and sends it to a server. It is also equipped with an emotion engine that analyzes the voice and facial expressions of workers through a facial recognition camera and microphone. The robot recognizes the worker's emotions in real time and sends them to the server along with the collected data.

[1410] The generative model is a machine learning algorithm that runs on a server, analyzes collected data, and generates production line optimization and improvement proposals. The generative model is implemented using machine learning frameworks such as Python and TensorFlow.

[1411] The emotion engine is also software that runs on the server and analyzes the voice and facial expression data sent from the robot to recognize the worker's emotions. The emotion engine uses technologies such as Emotion API and Facial Recognition API.

[1412] Program processing overview

[1413] 1. Data collection and transmission

[1414] The robots collect data in real time from sensors in the factory. The collected data is temporarily stored in a buffer and then sent to a server at regular intervals. The data is encrypted and transmitted securely.

[1415] 2. Data Analysis and Generation

[1416] The server analyzes the received data and extracts performance indicators for each production process. The generative model then generates optimization and improvement proposals for the production line based on the analyzed data and sends the results back to the server.

[1417] 3. Emotion recognition and suggestion adjustment

[1418] The robot collects the worker's voice and facial expressions, and the emotion engine analyzes them to recognize the worker's emotions. The emotion engine then sends the results of the recognition to the server, and the generative model adjusts the generated results based on them.

[1419] 4. Instruction Notification and Feedback

[1420] The server sends the generated improvement suggestions and instructions to the robot, which then presents them to the worker. The worker then takes action based on the suggestions. The robot then provides real-time feedback on the progress and results of the work to the server, which manages the entire system.

[1421] Examples of specific examples and prompts

[1422] Example 1: Worker A during a shift

[1423] If the emotion engine recognizes that worker A is tired, it will convey improvement suggestions obtained from the generative model (e.g., adjusting production speed) in more friendly terms.

[1424] Prompt: "What suggestions would be effective when Worker A is tired?"

[1425] Example 2: Troubleshooting a production line

[1426] In the event of a machine failure, the generative model suggests optimal troubleshooting steps and sends reassuring messages to the human worker via an emotion engine.

[1427] Prompt: "If a machine malfunction occurs, what troubleshooting steps and messages would be effective?"

[1428] As described above, the system of the present invention is capable of efficiently managing and improving a production line while taking into consideration the feelings of workers.

[1429] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1430] Step 1:

[1431] The robot (terminal) collects data in real time from each sensor in the factory. The input data is information about each process on the production line (e.g., temperature, pressure, speed, error rate, etc.). The robot temporarily stores this data in a buffer and performs the necessary preprocessing (e.g., format conversion, noise removal) to unify the data format. The preprocessed data is then stored in the buffer as output.

[1432] Step 2:

[1433] At regular intervals, the robot encrypts the temporarily stored data and sends it to the server. The input is the preprocessed data, and the output is the encrypted data. An encryption algorithm (e.g., AES encryption) is used for the processing. After encryption, the data is securely transferred to the server.

[1434] Step 3:

[1435] The server decrypts the received data and stores it in a database for analysis. The input is encrypted data and the output is decrypted data. A database management system (e.g., MySQL) is used to store the data efficiently.

[1436] Step 4:

[1437] The server analyzes the data stored in the database and extracts performance indicators for each production process. The input is the large amount of production data stored in the database, and the output is the performance indicators resulting from the analysis (e.g., lifting speed, error rate analysis results). Statistical analysis tools (e.g., Python's Pandas library) are used for data analysis.

[1438] Step 5:

[1439] The server requests the generative model to process data based on the performance indicators. The input is the analyzed performance indicators, and the output is the improvement proposals generated by the generative model. The generative model uses a machine learning algorithm (e.g., TensorFlow) to generate an optimal production line operation plan and improvement proposals (e.g., production speed adjustments, troubleshooting procedures).

[1440] Step 6:

[1441] The robot collects the worker's voice and facial expression data and sends it to the emotion engine. The input is the worker's real-time voice and facial expression data, and the output is the emotion recognition results (e.g., joy, anger, fatigue level). Emotion recognition uses voice recognition technology (e.g., Google Speech-to-Text) and image analysis technology (e.g., OpenCV).

[1442] Step 7:

[1443] The emotion engine analyzes voice and facial expression data to recognize the worker's emotions. The input is the voice and facial expression data sent from the robot, and the output is the emotion analysis results. In concrete terms, the voice data is converted into acoustic features, and the facial expression data is analyzed using image recognition technology. The recognized emotions are sent to the server.

[1444] Step 8:

[1445] The server adjusts the results of the generative model based on the sentiment analysis results received from the emotion engine. The inputs are the sentiment analysis results and the proposals generated by the generative model, and the output is the optimal proposal adjusted based on the sentiment. The server adjusts the tone and content of the proposal based on the sentiment analysis results and requests the generative model to process it again.

[1446] Step 9:

[1447] The server sends the adjusted improvement proposal to the robot, which then presents it to the worker. The input is the adjusted improvement proposal, and the output is information displayed in a presentation format (e.g., presentation text, voice message). Specifically, the robot presents information to the worker using a monitor or voice output device.

[1448] Step 10:

[1449] The worker takes action based on the proposed improvement suggestions. The input is the proposed suggestions, and the output is the worker's choice and the action to be taken (e.g., adjusting production speed, restarting the machine). The robot monitors the worker's choices and provides support as needed.

[1450] Step 11:

[1451] The robot sends feedback on the worker's progress and work results to the server in real time. The input is the worker's progress data, and the output is feedback information sent to the server. This allows the server to grasp the overall situation and make further improvement suggestions and optimizations.

[1452] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1453] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1454] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1455] [Fourth embodiment]

[1456] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1457] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1458] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1459] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1460] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1461] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1462] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1463] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1464] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1465] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1466] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1467] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1468] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1469] The system of the present invention includes a means for processing books using a generative model, a means for notifying the results generated by the generative model, a means for unifying notifications from multiple applications, and a means for optimizing user operations. The program processing flow of this system is explained in natural language, with specific examples.

[1470] System Overview

[1471] This system consists of a server, the user's smartphone (device), and a generative model. Notifications and operations of each application are centrally managed, and the generative model automatically handles the necessary operations.

[1472] Overview of the embodiment

[1473] User authentication and account information acquisition

[1474] When a user logs in to the app, the server calls the authentication API to authenticate the user. If authentication is successful, the server sends a request to the corresponding API to obtain account information for each of the user's applications, encrypts the account information returned from each application, and stores it in the database.

[1475] Retrieving application notifications and sending them to the server

[1476] The device receives notifications from each application installed on the device, temporarily stores the received notifications in a buffer, and encrypts and transmits the notification data in the buffer to a server at regular intervals.

[1477] Analyzing notifications and submitting them to a generated AI model

[1478] The server analyzes the received notification, identifies the notification type, and determines whether processing is necessary. Based on the analyzed notification, the server creates a request to the generative AI model for processing and sends the data to the generative AI model for analysis.

[1479] Processing notification content and generating instructions and replies

[1480] The generative model analyzes the data received from the server and generates appropriate summaries, instructions, and replies. For example, in the case of emails, it summarizes the text, and in the case of traffic information, it generates alternative transportation methods for cancelled services. The generated summaries and instructions are then sent back to the server.

[1481] Sending generated instructions and replies from the server to the device

[1482] The server organizes the summaries and instructions received from the generative model, encrypts them, and sends them to the device.

[1483] What to show the user and how to take action

[1484] The terminal decodes the summary and instructions received from the server and displays them to the user. The user then checks the displayed summary and instructions and decides on the next action. The terminal then receives the user's action and performs the necessary operation. After the operation is completed, it reports the result to the server.

[1485] Specific examples

[1486] For the Mail app

[1487] 1. The server obtains the user's email account information, encrypts it, and stores it.

[1488] 2. Your device will receive a notification of new email.

[1489] 3. The device sends the notification to the server.

[1490] 4. The server parses the email notification and sends the email body to the generative model.

[1491] 5. The generative model summarizes the email body and returns the summary results to the server.

[1492] 6. The server sends the summary results to the terminal.

[1493] 7. The terminal displays the summary to the user.

[1494] 8. The user reviews the summary and selects a reply.

[1495] 9. The device sends the user's reply via the email app.

[1496] 10. The terminal reports the completion of the reply to the server.

[1497] For traffic information apps

[1498] 1. The server obtains the user's traffic information account information, encrypts it, and stores it.

[1499] 2. The device receives a notification that the train is cancelled.

[1500] 3. The device sends the notification to the server.

[1501] 4. The server analyzes the cancellation notice and sends the cancellation information to the generative model.

[1502] 5. The generative model proposes alternative transportation options for the cancelled service and sends the results back to the server.

[1503] 6. The server sends the proposal results to the terminal.

[1504] 7. The device displays the proposed results to the user.

[1505] 8. The user reviews the proposal and selects an alternative.

[1506] 9. The terminal reserves alternative means based on the user's selection.

[1507] 10. The terminal reports the completion of the reservation to the server.

[1508] This system allows users to centrally manage notifications and their processing from various applications, significantly reducing the operational burden.

[1509] The processing flow will be explained below.

[1510] Step 1:

[1511] When a user logs in to an app, the server calls the authentication API to authenticate the user. If authentication is successful, the server sends a request to the corresponding API to obtain account information for each application of the user.

[1512] Step 2:

[1513] The server encrypts the account information returned by each application and stores it in a database, ensuring that user authentication information is stored securely.

[1514] Step 3:

[1515] The device receives notifications from each application installed on the device, such as new emails or train schedules.

[1516] Step 4:

[1517] The terminal temporarily stores the received notification in a buffer, thereby temporarily storing the notification information.

[1518] Step 5:

[1519] The terminal encrypts the notification data in the buffer at regular intervals and sends it to the server. Encryption ensures that the notification data is transferred safely.

[1520] Step 6:

[1521] The server analyzes the received notifications, identifies the notification type (e.g., email notification, traffic information notification), and determines the need for processing each notification.

[1522] Step 7:

[1523] When a notification is determined to require processing, the server creates a request to the generative model to request processing.

[1524] Step 8:

[1525] The server sends the notification details (e.g., email body, traffic information details) to the generative model.

[1526] Step 9:

[1527] The generative model analyzes the data received from the server and generates appropriate summaries, instructions, and replies. For example, in the case of emails, it summarizes the text, and in the case of traffic information, it generates alternative transportation options for cancelled services.

[1528] Step 10:

[1529] The generative model sends the generated summary and instructions back to the server.

[1530] Step 11:

[1531] The server organizes the summaries and instructions received from the generative model and sends them to the device. The data is encrypted before being transmitted.

[1532] Step 12:

[1533] The terminal decodes the summary and instructions received from the server and displays them to the user in a format that makes it easy for the user to understand the information.

[1534] Step 13:

[1535] The user reviews the displayed summary and instructions and decides on the next action (e.g., reply to email, select alternative transportation).

[1536] Step 14:

[1537] The device receives the user's action and performs the necessary operation, such as sending a reply to an email or booking transportation.

[1538] Step 15:

[1539] After the operation is completed, the terminal reports the result to the server, allowing the server to grasp the overall processing status.

[1540] Example 1

[1541] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1542] In modern society, users need to use multiple applications and receive various notifications from each. However, processing these notifications individually is cumbersome, time-consuming, and labor-intensive. There is also a risk of missing important notifications. Furthermore, optimizing operations across multiple applications is technically difficult, so an efficient means to reduce the burden on users is needed.

[1543] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1544] In this invention, the server includes means for performing user authentication, means for acquiring account information, means for receiving notifications from the terminal, means for analyzing the notifications, means for causing a generative AI model to process the notification content, means for organizing the generated results, means for transmitting the generated results to the terminal, means for receiving user operations, and means for executing processing based on the user operations. This allows the user to efficiently process centrally managed notifications and perform necessary operations quickly and automatically.

[1545] "User authentication" is a method of verifying the legitimacy of a user accessing a system.

[1546] "Account information" refers to the information required for a user to log in to each application, and typically includes a user name and password.

[1547] "Terminal" means a device operated by a user to receive and display notifications.

[1548] "Notifications" are information alerts or messages sent from each application to the terminal.

[1549] A "server" is a central computer system that handles the processing of the entire system.

[1550] A "generative AI model" is an artificial intelligence algorithm that analyzes the data it receives and generates appropriate summaries, instructions, and replies.

[1551] "Analysis of notifications" refers to the process of understanding the content of received notifications and determining the necessary processing.

[1552] "Result cleanup" is the process of formatting and organizing the results received from the generative model into an appropriate format before displaying them to the user.

[1553] "Performing an operation" is a means of carrying out a specific action based on a user's instructions.

[1554] "Centralized management" is a means of managing notifications and information from multiple applications through a unified interface.

[1555] The system of the present invention processes books using a generative AI model and notifies users of the results generated by the generative model. Specifically, it centrally manages notifications from multiple applications and optimizes user operation. This system consists of a server, a user's device such as a smartphone or tablet, and the generative AI model.

[1556] Hardware and software used

[1557] Server: Server for data processing and storage using cloud services (e.g., Amazon Web Services, Microsoft Azure)

[1558] Terminal: A device that functions as a user interface (e.g., a smartphone or tablet running iOS or Android).

[1559] Generative AI models: Natural language processing models (e.g., GPT-3, BERT)

[1560] User authentication and account information acquisition

[1561] The server authenticates the user using an authentication API (e.g. OAuth 2.0) based on the authentication information (email address and password) entered by the user.

[1562] If authentication is successful, the server uses each application's API to obtain account information, encrypts it with AES, and stores it in the database.

[1563] Retrieving application notifications and sending them to the server

[1564] The device receives notifications from each application that arrives to the user (stored in a temporary buffer), encrypts them with AES at regular intervals, and sends them to the server using the HTTPS protocol.

[1565] Analyzing notifications and submitting them to a generated AI model

[1566] The server analyzes the received notification and identifies the notification type (e.g., email or traffic information).

[1567] Once the required processing is determined, the server makes a request to the generative AI model, sending the data along with a prompt such as "Please summarize the contents of this email."

[1568] Processing notification content and generating instructions and replies

[1569] The generative AI model analyzes the data received from the server and performs the specified task (for example, summarizing emails or suggesting alternative transportation options in response to service suspension information).

[1570] The generated results are sent back to the server.

[1571] Sending generated instructions and replies from the server to the device

[1572] The server organizes the results received from the generative AI model and formats them in a format that is easy for the user to view.

[1573] It is then AES encrypted and sent to the terminal via HTTPS protocol.

[1574] What to show the user and how to take action

[1575] The device decodes the data received from the server and displays it to the user.

[1576] The user checks the displayed data and selects the next action.

[1577] Based on the user's instructions, the terminal performs the necessary operations (for example, replying to an email or booking transportation) and reports the results to the server.

[1578] For example, the following prompts may be used:

[1579] Please summarize the contents of the email.

[1580] "Please suggest alternative transportation for this suspension."

[1581] The system allows users to centralize notifications from multiple applications and respond quickly and efficiently with the help of generative AI models.

[1582] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1583] Step 1:

[1584] User authentication and account information acquisition

[1585] The user enters their email address and password to log in to the app.

[1586] The server calls an authentication API (e.g., OAuth 2.0) based on the entered authentication information and verifies the user's validity.

[1587] If authentication is successful, the server uses each application's API to obtain the user's account information.

[1588] Input: User's email address and password.

[1589] Output: An authentication token and the user's account information (e.g., Google Calendar and Gmail information).

[1590] Specific operation: The server encrypts the acquired account information using AES and stores it in the database.

[1591] Step 2:

[1592] Retrieving application notifications and sending them to the server

[1593] The device receives notifications from each installed application.

[1594] Received notifications are temporarily stored in a buffer.

[1595] At regular intervals, the terminal encrypts the notification data in the buffer using AES and sends it to the server using HTTPS.

[1596] Input: New notifications from each application.

[1597] Output: Encrypted notification data.

[1598] Specific operation: When the device receives a notification of a new email, it temporarily stores the content in a buffer and sends it to the server every 10 minutes.

[1599] Step 3:

[1600] Analyzing notifications and submitting them to a generated AI model

[1601] The server analyzes the received notification and identifies the notification type (e.g., email, traffic information).

[1602] Once the required processing is determined, the server makes a request to the generative AI model.

[1603] For example, the server may send the email body along with a prompt saying "Please summarize the contents of this email."

[1604] Input: Encrypted notification data.

[1605] Output: Parsing requests to the generative AI model.

[1606] What it does: The server parses the email notification and sends the email body along with a prompt to the generative AI model.

[1607] Step 4:

[1608] Processing notification content and generating instructions and replies

[1609] The generative AI model analyzes the notification data received from the server.

[1610] Based on the analysis results, the generative AI model generates summaries, instructions, and replies.

[1611] Input: Analysis request and notification data sent by the server.

[1612] Output: Generated summary, instructions and reply.

[1613] Specific operation: The generative AI model extracts important points from the received email text, generates a summary, and sends the summary back to the server.

[1614] Step 5:

[1615] Sending generated instructions and replies from the server to the device

[1616] The server organizes and formats the summaries and instructions received from the generative AI model.

[1617] They are AES encrypted and sent to the terminal using HTTPS.

[1618] Input: The summary or instructions sent from the generative AI model.

[1619] Output: Encrypted summary or instructions.

[1620] Specific operation: The server formats the summary of the email received from the generated AI model, encrypts it with AES, and sends it to the terminal.

[1621] Step 6:

[1622] What to show the user and how to take action

[1623] The terminal decodes the summary and instructions received from the server and displays them to the user.

[1624] The user checks the displayed summary and instructions and selects the next action.

[1625] Based on the user's instructions, the device performs the necessary operations (e.g., replying to emails or booking transportation) and reports the results to the server.

[1626] Input: Encrypted summary or instructions.

[1627] Output: The user's actions and their results.

[1628] Specific operation: The device decodes the summary received from the server and displays it to the user. After the user presses the reply button, the reply is sent via the email app and the completion is notified to the server.

[1629] (Application example 1)

[1630] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1631] In electronic payment services, it has been difficult to efficiently manage information such as payments, bills, and transfers sent from multiple applications and provide optimal information and operational support to users. Furthermore, the sheer volume of notifications means users run the risk of missing important notifications, making it difficult to respond quickly and appropriately. This results in delays in users' responses to important payments and bills, increasing the likelihood of financial trouble.

[1632] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1633] In this invention, the server includes means for processing books using a generative model, means for notifying results generated by the generative model, means for centrally managing notifications from multiple applications, means for optimizing user operations, and means for centrally managing electronic payment notifications and summarizing and organizing notifications of various payments, bills, and transfers. This allows for efficient processing of a large number of notifications related to electronic payments, enabling users to respond quickly and optimally without missing important information.

[1634] A "generative model" is a model that uses machine learning or deep learning to generate, analyze, or predict data.

[1635] "Books" include information such as documents, papers, e-mails, and traffic information.

[1636] A "notification" is a message that conveys information from each application or system to the user.

[1637] "Centralized management" refers to the aggregation of data and notifications from multiple sources and the handling of them in a unified manner.

[1638] "Optimizing user operations" means enabling the user to complete the operations they need to perform easily and quickly.

[1639] "Electronic Payment Notice" means a notice containing payment, billing, transfer, or other information relating to an Electronic Payment Service.

[1640] A summary is a short summary of detailed information that extracts only the most important points.

[1641] "Organizing" means arranging different information and data in a unified format and order.

[1642] "Information provision" refers to conveying necessary information to users in an appropriate format.

[1643] "Operation support" refers to supporting the operations performed by the user and helping them to proceed efficiently.

[1644] This invention is a system that uses a generative model to efficiently manage electronic payment notifications and provide optimal information to users. This system consists of a server, a user terminal, and a generative AI model.

[1645] When a user logs in to the application, the server authenticates the user using the authentication API. If user authentication is successful, the server encrypts and stores the electronic payment account information in a database. This database is used to store the data required for subsequent notification processing.

[1646] The terminal consists of the user's smartphone or smart glasses and receives notifications from each electronic payment application. The notifications are temporarily stored in a buffer on the terminal and encrypted and sent to the server at regular intervals.

[1647] The server analyzes the notification data received from the device and requests processing from the generative AI model. The generative model generates an appropriate summary and instructions from the analyzed notification content and returns the results to the server. The server organizes the summary and instructions received from the generative model and sends them to the user's device.

[1648] The user's device decodes the summary and instructions received from the server and displays them to the user. The user then checks the displayed summary and instructions and selects the next action. Based on the selected action, the device performs the necessary operation and reports the result to the server after the operation is completed.

[1649] Hardware and software used

[1650] Hardware: User's smartphone, smart glasses

[1651] software:

[1652] Authentication API: A service for user authentication

[1653] Electronic Payment API: Acquisition and notification of electronic payment account information

[1654] Generative AI model: A model service that summarizes notifications and generates optimal instructions

[1655] Encryption Library: Uses Python's cryptography library to encrypt data.

[1656] Specific examples

[1657] When a user uses multiple electronic payment services, they frequently receive payment notices, billing notices, transfer notices, etc. from each service. Managing all of these notices individually can be cumbersome, so by using this system, they can be managed centrally.

[1658] For example, if a user receives a payment notification like this:

[1659] Notification: "Your account has been charged $10,000. Your balance is $50,000."

[1660] When this notification arrives on the device, the notification data is sent to the server, and the generative AI model generates a summary such as, "A payment of ¥10,000 has been made. The balance is ¥50,000."

[1661] Example prompts to send to the generative AI model:

[1662] Parse the notification and generate a summary result with appropriate instructions:

[1663] Notification: "Your account has been charged $10,000. Your balance is $50,000."

[1664] By displaying such summary results to the user, the user can quickly check important information without missing anything, and the necessary next steps are presented, optimizing the user's operations, thereby reducing the risk of financial trouble and improving management efficiency.

[1665] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1666] Step 1:

[1667] When a user logs in to an application, the server authenticates the user using the authentication API. When the user enters their login information (username, password), the authentication API sends that information to the server, and if authentication is successful, an access token is issued. The server receives this access token and confirms that user authentication was successful.

[1668] Step 2:

[1669] The server sends a request to the electronic payment API to get electronic payment account information using the access token. This request includes the user ID and the access token. The electronic payment API returns the corresponding account information, and the server stores the account information in a database using an encryption library.

[1670] Step 3:

[1671] The terminal receives notifications from each electronic payment application using the user's smartphone or smart glasses. When a notification arrives, the terminal temporarily stores the notification in a buffer, and then encrypts the notification data in the buffer and sends it to the server at regular intervals. This encryption is performed using the encryption library mentioned above.

[1672] Step 4:

[1673] The server receives and decodes the notification data sent from the device, analyzes the notification, and creates a request to the generative AI model based on its content. Specifically, it extracts the important parts of the notification to generate a prompt sentence and sends it to the generative AI model service.

[1674] Step 5:

[1675] The generative AI model summarizes the notification content based on the received prompt and generates appropriate instructions. For example, if the notification is "Your account has been paid ¥10,000. The balance is ¥50,000," the generative AI model generates a summary of "Your account has been paid ¥10,000. The balance is ¥50,000" and an instruction to "Check your new income."

[1676] Step 6:

[1677] The server receives the summary and instructions returned by the generative AI model, organizes and encrypts them, and sends the organized data to the user's device, ensuring the integrity and security of the processing results.

[1678] Step 7:

[1679] The terminal decodes the summary and instructions sent by the server and displays them to the user, who then reviews the displayed information and selects the next action to take, such as confirming payment details or approving the charge.

[1680] Step 8:

[1681] The terminal performs the corresponding operation based on the user's selection. For example, if payment confirmation is required, the terminal performs a confirmation operation through the electronic payment application. After this, the terminal reports the operation result to the server and updates the database if necessary.

[1682] This completes the entire flow, allowing the user to centrally manage multiple electronic payment notifications and respond quickly and appropriately.

[1683] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1684] The system of the present invention includes a means for processing books using a generative model and notifying the results generated by the generative model, a means for unifying and managing notifications from multiple applications, a means for optimizing user operations, and an emotion engine for recognizing user emotions. The program processing flow of this system is explained in natural language, with specific examples.

[1685] System Overview

[1686] This system consists of a server, the user's smartphone (device), a generative model, and an emotion engine. It centrally manages notifications and operations for each application, and the generative model automatically processes the necessary operations. By incorporating the emotion engine, it is possible to provide optimal responses according to the user's emotions.

[1687] Overview of the embodiment

[1688] User authentication and account information acquisition

[1689] When a user logs in to an app, the server calls the authentication API to authenticate the user. If authentication is successful, the server sends a request to the corresponding API to obtain account information for each application of the user.

[1690] The server encrypts the account information returned from each application and stores it in a database.

[1691] Retrieving application notifications and sending them to the server

[1692] The device receives notifications from each application installed on the device, temporarily stores the received notifications in a buffer, and encrypts and transmits the notification data in the buffer to a server at regular intervals.

[1693] Parsing notifications and submitting generative models

[1694] The server analyzes the received notifications, identifies the notification type (e.g., email notification, traffic information notification), and determines the need for processing for each. Based on the analyzed notification, it creates a request to the Generative Model for processing and sends the data to be analyzed to the Generative Model.

[1695] Processing notification content and generating instructions and replies

[1696] The generative model analyzes the data received from the server and generates appropriate summaries, instructions, and replies. For example, in the case of emails, it summarizes the text, and in the case of traffic information, it generates alternative transportation methods for cancelled services. The generated summaries and instructions are then sent back to the server.

[1697] Emotion recognition and generative modeling results adjustment

[1698] The device uses an emotion engine to recognize the user's emotions from the user's voice, facial expressions, or text. For example, it uses voice recognition or image recognition to detect the user's current emotions (e.g., joy, sadness, anger, surprise, etc.).

[1699] The generative model adjusts its output depending on the perceived user emotion: for example, if the user is stressed, it can adjust the reply to a calmer, more gentle tone.

[1700] Sending generated instructions and replies from the server to the device

[1701] The server organizes the summaries and instructions received from the generative model and emotion engine, encrypts them, and transmits them to the terminal.

[1702] What to show the user and how to take action

[1703] The terminal decodes the summary and instructions received from the server and displays them to the user in a format that makes it easy for the user to understand the information.

[1704] The user reviews the displayed summary and instructions and decides on the next action (e.g., reply to email, select alternative transportation).

[1705] The device receives the user's action and performs the necessary operation, such as sending a reply to an email or booking transportation.

[1706] After the operation is completed, the terminal reports the result to the server, allowing the server to grasp the overall processing status.

[1707] Specific examples

[1708] For the Mail app

[1709] 1. The server obtains the user's email account information, encrypts it, and stores it.

[1710] 2. Your device will receive a notification of new email.

[1711] 3. The device sends the notification to the server.

[1712] 4. The server parses the email notification and sends the email body to the generative model.

[1713] 5. The generative model summarizes the email body and returns the summary results to the server.

[1714] 6. The device recognizes the user's emotions using an emotion engine (e.g., if the user is feeling stressed).

[1715] 7. The generative model adjusts the reply based on the perceived sentiment.

[1716] 8. The server sends the summary results and the adjusted reply to the terminal.

[1717] 9. The terminal displays the summary to the user and prompts them to reply.

[1718] 10. The user selects a reply and the device sends the reply via the email app.

[1719] 11. The terminal reports the completion of the reply to the server.

[1720] For traffic information apps

[1721] 1. The server obtains the user's traffic information account information, encrypts it, and stores it.

[1722] 2. The device receives a notification that the train is cancelled.

[1723] 3. The device sends the notification to the server.

[1724] 4. The server analyzes the cancellation notice and sends the cancellation information to the generative model.

[1725] 5. The generative model proposes alternative transportation options for the cancelled service and sends the results back to the server.

[1726] 6. The device recognizes the user's emotions using an emotion engine (e.g., if the user is anxious).

[1727] 7. The generative model adjusts the alternative transportation suggestions based on the perceived sentiment.

[1728] 8. The server sends the proposal results to the terminal.

[1729] 9. The device displays the proposed results to the user and encourages them to take action.

[1730] 10. The user reviews the suggestions and selects an alternative.

[1731] 11. The terminal reserves the alternative means and reports the completion of the reservation to the server.

[1732] This system not only allows users to centrally manage notifications and their handling from various applications, but also, by combining it with an emotion engine, enables optimal responses that take into consideration the user's emotions. This system reduces the operational burden and can provide more personal and effective support.

[1733] The processing flow will be explained below.

[1734] Step 1:

[1735] When a user logs in to an app, the server calls the authentication API to authenticate the user. If authentication is successful, the server sends a request to the corresponding API to obtain account information for each application of the user.

[1736] Step 2:

[1737] The server encrypts the account information returned by each application and stores it in a database, ensuring that user authentication information is stored securely.

[1738] Step 3:

[1739] The device receives notifications from each application installed on the device, such as new emails or train schedules.

[1740] Step 4:

[1741] The terminal temporarily stores the received notification in a buffer, thereby temporarily storing the notification information.

[1742] Step 5:

[1743] The terminal encrypts the notification data in the buffer at regular intervals and sends it to the server. Encryption ensures that the notification data is transferred safely.

[1744] Step 6:

[1745] The server analyzes the received notifications, identifies the notification type (e.g., email notification, traffic information notification), and determines the need for processing each notification.

[1746] Step 7:

[1747] When a notification is determined to require processing, the server creates a request to the generative model to request processing.

[1748] Step 8:

[1749] The server sends the notification details (e.g., email body, traffic information details) to the generative model.

[1750] Step 9:

[1751] The generative model analyzes the data received from the server and generates appropriate summaries, instructions, and replies. For example, in the case of emails, it summarizes the text, and in the case of traffic information, it generates alternative transportation options for cancelled services.

[1752] Step 10:

[1753] The generative model sends the generated summary and instructions back to the server.

[1754] Step 11:

[1755] The terminal uses an emotion engine to recognize the user's emotion from the user's voice, facial expression, or text, using, for example, voice recognition, image recognition, and text analysis.

[1756] Step 12:

[1757] The emotion engine identifies the user's emotions and feeds the results back to the generative model. For example, if the user is feeling stressed, the generative model will adjust its results based on that information.

[1758] Step 13:

[1759] The generative model receives feedback from the emotion engine and adjusts the generated summaries and replies, allowing it to provide content that takes the user's emotions into consideration.

[1760] Step 14:

[1761] The server organizes summaries and instructions received from the generative model and emotion engine, and sends them to the device. The data is encrypted before transmission.

[1762] Step 15:

[1763] The terminal decodes the summary and instructions received from the server and displays them to the user in a format that makes it easy for the user to understand the information.

[1764] Step 16:

[1765] The user reviews the displayed summary and instructions and decides on the next action (e.g., reply to email, select alternative transportation).

[1766] Step 17:

[1767] The device receives the user's action and performs the necessary operation, such as sending a reply to an email or booking transportation.

[1768] Step 18:

[1769] After the operation is completed, the terminal reports the result to the server, allowing the server to grasp the overall processing status.

[1770] Example 2

[1771] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1772] Conventional notification management systems handle and manage notifications from multiple applications individually, making user operations cumbersome. Furthermore, they handle notifications uniformly without considering the user's feelings, making it difficult to improve the user experience. Furthermore, the analysis results and responses to notification content were sometimes inappropriate, placing a burden on users.

[1773] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1774] In this invention, the server includes a means for analyzing and processing notifications using a generative model, a means for temporarily storing the analyzed notification content and encrypting and transmitting it at regular intervals, and a means for centrally managing notifications from multiple applications. This simplifies user operations and improves the accuracy of notification content. Furthermore, by combining a means for recognizing emotions from the user's voice, facial expression, or text and a means for adjusting the generation results based on the recognized emotions, the burden of work can be reduced and a personalized and appropriate response can be provided to the user.

[1775] A "generative model" is an algorithm that uses artificial intelligence techniques to analyze input data and generate specific responses or text.

[1776] "Notification" refers to information or alerts provided by each application to the user's device.

[1777] "Encryption" is the process of transforming data into a form that cannot be understood by third parties in order to ensure its security.

[1778] "Emotion" refers to a psychological state determined by analyzing the user's voice, facial expression, text, etc.

[1779] "Centralized management" refers to the centralized management of multiple different pieces of information and processes on a single system or platform.

[1780] "Analysis" is the process of breaking down input data or information and interpreting its meaning.

[1781] "Means" are methods or tools used to achieve a particular function or purpose.

[1782] A "notification buffer" is an area or memory for temporarily storing received notifications.

[1783] "Temporary archiving" is the process of storing data for a short period of time for later processing.

[1784] Our system uses generative models to analyze and process notifications, providing a unified mechanism for managing notifications across multiple applications, streamlining user interaction and enabling personalized responses based on emotion recognition.

[1785] System Configuration

[1786] The system consists of the following components:

[1787] server

[1788] User's smartphone (device)

[1789] Generative Model

[1790] Emotion Engine

[1791] Hardware and Software Details

[1792] server

[1793] The server performs tasks such as user authentication, analysis of notification data, sending requests to the generative model, receiving data from the emotion engine, and managing the database.

[1794] Hardware: A server with a powerful CPU, sufficient memory, and storage

[1795] Software: authentication systems, API request handlers, databases (e.g., MySQL), cryptography libraries (e.g., OpenSSL)

[1796] Terminal

[1797] The device refers to the user's smartphone or tablet, which temporarily stores notification data from installed applications and sends it to a server.

[1798] Hardware: Smartphones, tablets

[1799] Software: Applications (e.g., Gmail app, Google Maps app), notification buffer, SSL / TLS communication

[1800] Generative Model

[1801] The generative model analyzes notification data and generates responses, for example, by using natural language processing (NLP) techniques to generate a summary of the notification content and an appropriate reply.

[1802] Software: OpenAI's GPT-4, etc.

[1803] Emotion Engine

[1804] The emotion engine is responsible for recognizing emotions from the user's voice, facial expressions, text, etc., and providing the results to the generative model.

[1805] Software: Google Speech-to-Text, Amazon Transcribe, OpenCV, Google Cloud Vision API

[1806] Data processing and calculation

[1807] User authentication and account information acquisition

[1808] When a user logs in, the server authenticates the user using the authentication API and obtains the account information for each application of the authenticated user. This information is protected by AES encryption and stored in the database.

[1809] Getting notifications

[1810] The device receives notifications from each application on the device, temporarily stores them in a buffer, and then encrypts the notification data in the buffer and sends it to the server at regular intervals.

[1811] Parsing notifications and processing generative models

[1812] The server parses the notification, identifies the notification type, and then sends a request to the generative model, which then analyzes the notification content and generates an appropriate summary or reply message, which is then sent back to the server.

[1813] Emotion recognition and generation adjustment

[1814] The device recognizes emotions from the user's voice, facial expressions, or text and sends the results to the server, which then integrates data from the generative model and emotion engine to generate an optimized response based on the emotion.

[1815] Specific examples

[1816] For the Mail app

[1817] 1. The server obtains the user's email account information, encrypts it, and stores it.

[1818] 2. Your device will receive a notification of new email.

[1819] 3. The device sends the notification to the server.

[1820] 4. The server parses the email notification and sends the email body to the generative model.

[1821] 5. The generative model summarizes the email body and returns the summary results to the server.

[1822] 6. The device recognizes the user's emotions using an emotion engine.

[1823] 7. The generative model adjusts the reply based on the perceived sentiment.

[1824] 8. The server sends the summary results and the adjusted reply to the terminal.

[1825] 9. The terminal displays the summary to the user and prompts them to reply.

[1826] 10. The user selects a reply and the device sends the reply via the email app.

[1827] 11. The terminal reports the completion of the reply to the server.

[1828] For traffic information apps

[1829] 1. The server obtains the user's traffic information account information, encrypts it, and stores it.

[1830] 2. The device receives a notification that the train is cancelled.

[1831] 3. The device sends the notification to the server.

[1832] 4. The server analyzes the cancellation notice and sends the cancellation information to the generative model.

[1833] 5. The generative model proposes alternative transportation options for the cancelled service and sends the results back to the server.

[1834] 6. The device recognizes the user's emotions using an emotion engine.

[1835] 7. The generative model adjusts the alternative transportation suggestions based on the perceived sentiment.

[1836] 8. The server sends the proposal results to the terminal.

[1837] 9. The device displays the proposed results to the user and encourages them to take action.

[1838] 10. The user reviews the suggestions and selects an alternative.

[1839] 11. The terminal reserves the alternative means and reports the completion of the reservation to the server.

[1840] As a result, this system centralizes the notifications a user receives from various applications and uses a generative model and emotion engine to automatically provide appropriate and personalized responses.

[1841] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1842] Step 1: User authentication and account information acquisition

[1843] When a user logs in to an app, the server accesses the authentication API to authenticate the user. The server receives the user ID and password as input. If authentication is successful, an access token is generated (operation: authentication process). The server then sends a request to the corresponding API to obtain account information for each application of the user (operation: API call). The output is the account information for each application.

[1844] Step 2: Save the account information you obtained

[1845] The server protects the account information returned from each application (e.g., user ID, access token, application setting information) with AES encryption and stores it securely in the database (operation: encryption processing and database storage). The input is the account information of each application, and the output is encrypted account information.

[1846] Step 3: Get notifications for your application

[1847] The device receives notifications from each installed application. Specifically, notifications are received through the OS notification center (Action: Receive Notification). The input is the raw notification from the application, and the output is the notification data stored in the buffer.

[1848] Step 4: Temporarily store and send notification data

[1849] The terminal temporarily stores the acquired notification data in a buffer, and at regular intervals (e.g., every minute), it AES-encrypts the notification data in the buffer and sends it to the server (operation: buffering, encryption, communication). The input is the notification data in the buffer, and the output is the encrypted notification data.

[1850] Step 5: Parse the notification

[1851] The server decrypts the notification data received from the device and analyzes it using natural language processing (NLP) technology to identify the notification type (e.g., email notification, traffic information notification) (operation: data analysis and classification). The input is the encrypted notification data, and the output is the classified notification data and the analysis results.

[1852] Step 6: Submitting the Generative Model

[1853] The server creates a processing request to the generative model based on the analyzed notification data and sends the data (action: create request and send). The input is the analyzed notification data and its classification results, and the output is the request data sent to the generative model.

[1854] Step 7: Response generation using the generative model

[1855] The generative model analyzes notification data received from the server and generates appropriate summaries, instructions, and replies (operations: notification analysis and text generation). Specific operations include using NLP algorithms. The input is notification data from the server, and the output is the generated summary or reply.

[1856] Step 8: Sending the Generated Response Back to the Server

[1857] The generative model returns the generated summary and reply text to the server (action: data transmission). The input is the generated summary and reply text, and the output is the response data received on the server side.

[1858] Step 9: Emotion Recognition

[1859] The device uses an emotion engine to recognize the user's emotions from the user's voice, facial expression, or text (calculation: voice recognition, image recognition). It works by acquiring real-time data using a microphone or camera and performing emotion analysis. The input is the user's voice, facial expression, and text, and the output is recognized emotional data.

[1860] Step 10: Adjusting the results

[1861] The generative model adjusts the generated results based on the recognized user emotion (operation: adjust text tone). The operation includes adjusting the generated text (e.g., changing the tone to a gentler tone). The input is the recognized emotion data and the generated results, and the output is the adjusted response data.

[1862] Step 11: Sending the generated instruction / reply

[1863] The server organizes the summaries and instructions received from the generative model and emotion engine, encrypts them, and sends them to the terminal (operations: data organization, encryption, communication). The input is the adjusted response data and emotion data, and the output is the encrypted data sent to the terminal.

[1864] Step 12: Show the user and take action

[1865] The terminal decodes the summary or instructions received from the server and displays them to the user (action: decode and display). The input is the encrypted response data, and the output is the decoded instructions or summary.

[1866] The user checks the displayed summary and instructions and decides on the next action (action: action selection). The input is the decoded instructions and summary, and the output is the user's selected action.

[1867] The terminal receives the user's action and executes the necessary operation (action: sending a reply or making a reservation). The input is the user's selected action, and the output is the execution result.

[1868] Step 13: Reporting the results

[1869] After the operation is completed, the terminal reports the result to the server (operation: communication and log storage). The input is the execution result of the operation, and the output is the result data reported to the server.

[1870] (Application example 2)

[1871] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1872] In recent years, there has been a demand for efficient management and next-generation work environments on production lines within factories. Furthermore, productivity can be affected by the emotions of workers, so it is important to respond in a way that takes their emotions into consideration. Conventional systems have struggled to recognize workers' emotions in real time, analyze production line data, and propose improvements. This has led to issues such as insufficient optimization of production efficiency and reduction of worker burden.

[1873] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for processing books using a generative model, a means for notifying results generated by the generative model, a means for centrally managing notifications from multiple applications, a means for optimizing user operations, a means for recognizing worker emotions using an emotion engine, a means for adjusting generated results based on the worker emotions, a means for collecting data in real time from devices in the factory, and a means for analyzing the collected data and generating improvement proposals. This enables efficient management of the production line and improvement proposals while taking into consideration the emotions of workers.

[1874] A "generative model" is a machine learning algorithm for generating specific output data from input data.

[1875] "Book" means a document or book containing information in digital form.

[1876] "Notification" means the provision of information from the system to the user.

[1877] "Centralized management" refers to the centralized management of multiple information sources and operations through a single interface.

[1878] The "means for optimizing user operations" is a function for making user operations more efficient and comfortable.

[1879] An "emotion engine" is a system that recognizes a user's emotions by analyzing voice, facial expressions, or text.

[1880] "Means for adjusting generated results based on the worker's emotions" refers to a function that changes the system's responses and suggestions depending on the worker's recognized emotions.

[1881] "Factory equipment" refers to hardware and sensors used on production lines and in manufacturing processes.

[1882] "Means of collecting data in real time" refers to the ability to instantly obtain ongoing data.

[1883] "Means for generating improvement proposals" is a function that generates specific proposals for improving productivity from the results of data analysis.

[1884] System Overview

[1885] The system of the present invention combines a generative model and an emotion engine to efficiently and optimally manage production lines in factories. The system consists of a server, a robot (terminal), a generative model, and an emotion engine. Data is collected in real time from each device, and the generative model analyzes it to generate improvement suggestions and provides instructions to workers via the robot. The emotion engine also recognizes the worker's emotions and adjusts the generated results based on them.

[1886] Hardware and Software

[1887] The server is responsible for centralized data management and analysis. The software running on the server includes a data collection API, data processing scripts, and generative models. The server is equipped with a high-performance database system running Linux and a powerful processor.

[1888] The robot (terminal) collects data from sensors in the factory and sends it to a server. It is also equipped with an emotion engine that analyzes the voice and facial expressions of workers through a facial recognition camera and microphone. The robot recognizes the worker's emotions in real time and sends them to the server along with the collected data.

[1889] The generative model is a machine learning algorithm that runs on a server, analyzes collected data, and generates production line optimization and improvement proposals. The generative model is implemented using machine learning frameworks such as Python and TensorFlow.

[1890] The emotion engine is also software that runs on the server and analyzes the voice and facial expression data sent from the robot to recognize the worker's emotions. The emotion engine uses technologies such as Emotion API and Facial Recognition API.

[1891] Program processing overview

[1892] 1. Data collection and transmission

[1893] The robots collect data in real time from sensors in the factory. The collected data is temporarily stored in a buffer and then sent to a server at regular intervals. The data is encrypted and transmitted securely.

[1894] 2. Data Analysis and Generation

[1895] The server analyzes the received data and extracts performance indicators for each production process. The generative model then generates optimization and improvement proposals for the production line based on the analyzed data and sends the results back to the server.

[1896] 3. Emotion recognition and suggestion adjustment

[1897] The robot collects the worker's voice and facial expressions, and the emotion engine analyzes them to recognize the worker's emotions. The emotion engine then sends the results of the recognition to the server, and the generative model adjusts the generated results based on them.

[1898] 4. Instruction Notification and Feedback

[1899] The server sends the generated improvement suggestions and instructions to the robot, which then presents them to the worker. The worker then takes action based on the suggestions. The robot then provides real-time feedback on the progress and results of the work to the server, which manages the entire system.

[1900] Examples of specific examples and prompts

[1901] Example 1: Worker A during a shift

[1902] If the emotion engine recognizes that worker A is tired, it will convey improvement suggestions obtained from the generative model (e.g., adjusting production speed) in more friendly terms.

[1903] Prompt: "What suggestions would be effective when Worker A is tired?"

[1904] Example 2: Troubleshooting a production line

[1905] In the event of a machine failure, the generative model suggests optimal troubleshooting steps and sends reassuring messages to the human worker via an emotion engine.

[1906] Prompt: "If a machine malfunction occurs, what troubleshooting steps and messages would be effective?"

[1907] As described above, the system of the present invention is capable of efficiently managing and improving a production line while taking into consideration the feelings of workers.

[1908] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1909] Step 1:

[1910] The robot (terminal) collects data in real time from each sensor in the factory. The input data is information about each process on the production line (e.g., temperature, pressure, speed, error rate, etc.). The robot temporarily stores this data in a buffer and performs the necessary preprocessing (e.g., format conversion, noise removal) to unify the data format. The preprocessed data is then stored in the buffer as output.

[1911] Step 2:

[1912] At regular intervals, the robot encrypts the temporarily stored data and sends it to the server. The input is the preprocessed data, and the output is the encrypted data. An encryption algorithm (e.g., AES encryption) is used for the processing. After encryption, the data is securely transferred to the server.

[1913] Step 3:

[1914] The server decrypts the received data and stores it in a database for analysis. The input is encrypted data and the output is decrypted data. A database management system (e.g., MySQL) is used to store the data efficiently.

[1915] Step 4:

[1916] The server analyzes the data stored in the database and extracts performance indicators for each production process. The input is the large amount of production data stored in the database, and the output is the performance indicators resulting from the analysis (e.g., lifting speed, error rate analysis results). Statistical analysis tools (e.g., Python's Pandas library) are used for data analysis.

[1917] Step 5:

[1918] The server requests the generative model to process data based on the performance indicators. The input is the analyzed performance indicators, and the output is the improvement proposals generated by the generative model. The generative model uses a machine learning algorithm (e.g., TensorFlow) to generate an optimal production line operation plan and improvement proposals (e.g., production speed adjustments, troubleshooting procedures).

[1919] Step 6:

[1920] The robot collects the worker's voice and facial expression data and sends it to the emotion engine. The input is the worker's real-time voice and facial expression data, and the output is the emotion recognition results (e.g., joy, anger, fatigue level). Emotion recognition uses voice recognition technology (e.g., Google Speech-to-Text) and image analysis technology (e.g., OpenCV).

[1921] Step 7:

[1922] The emotion engine analyzes voice and facial expression data to recognize the worker's emotions. The input is the voice and facial expression data sent from the robot, and the output is the emotion analysis results. In concrete terms, the voice data is converted into acoustic features, and the facial expression data is analyzed using image recognition technology. The recognized emotions are sent to the server.

[1923] Step 8:

[1924] The server adjusts the results of the generative model based on the sentiment analysis results received from the emotion engine. The inputs are the sentiment analysis results and the proposals generated by the generative model, and the output is the optimal proposal adjusted based on the sentiment. The server adjusts the tone and content of the proposal based on the sentiment analysis results and requests the generative model to process it again.

[1925] Step 9:

[1926] The server sends the adjusted improvement proposal to the robot, which then presents it to the worker. The input is the adjusted improvement proposal, and the output is information displayed in a presentation format (e.g., presentation text, voice message). Specifically, the robot presents information to the worker using a monitor or voice output device.

[1927] Step 10:

[1928] The worker takes action based on the proposed improvement suggestions. The input is the proposed suggestions, and the output is the worker's choice and the action to be taken (e.g., adjusting production speed, restarting the machine). The robot monitors the worker's choices and provides support as needed.

[1929] Step 11:

[1930] The robot sends feedback on the worker's progress and work results to the server in real time. The input is the worker's progress data, and the output is feedback information sent to the server. This allows the server to grasp the overall situation and make further improvement suggestions and optimizations.

[1931] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1932] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1933] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1934] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1935] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1936] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1937] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1938] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1939] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1940] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1941] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1942] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1943] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1944] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1945] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1946] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory...

Claims

1. means for processing the book using the generative model; a means for communicating the results generated by the generative model; A way to centrally manage notifications from multiple applications, means for optimizing user interaction; A system including:

2. 2. The system of claim 1, wherein the book is an email.

3. 2. The system according to claim 1, wherein the book is traffic information.

Images