System

Abstract

A system is provided.SOLUTION: The system includes a means for analyzing an e-mail and automatically generating a summary of contents by using a generated AI, a means for determining the importance of the e-mail, a means for displaying the summary on the basis of the determined importance, a means for predicting an action to be taken next by referring to a past correspondence history and generating an action list, and a means for displaying the action list and the summary on a user terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] For business people who receive a large volume of e-mail every day, important e-mails often get buried under other e-mails. As a result, there is an increased risk that important work instructions or customer communications will be delayed or overlooked. This reduces work efficiency and causes problems such as failure to keep up with important communications. The present invention aims to reduce the risk of such e-mail omissions and delayed responses, thereby improving work efficiency. [Means for solving the problem]

[0005] To solve this problem, the present invention provides an email analysis and summary generation system using generative AI, which includes the following means:

[0006] 1. A method of using generative AI to analyze emails and automatically generate summaries of their contents.

[0007] 2. A means of determining the importance of an email.

[0008] 3. A means of displaying summaries based on determined importance.

[0009] 4. A way to refer to past response history to predict what actions to take next and create an action list.

[0010] 5. A means for displaying the action list and summary on the user's terminal.

[0011] This allows users to respond quickly and accurately without important emails getting buried under other emails. Furthermore, the system automatically retrieves received emails and uses metadata such as the sender and subject to determine their importance, improving the efficiency and accuracy of the entire system.

[0012] "Generative AI" is a technology that uses artificial intelligence to analyze input data and automatically summarize or generate its content.

[0013] "Email" is a means of communication that contains text and attachments sent and received over the Internet or other networks.

[0014] The "summary" is information that summarizes the contents of an email by analyzing the entire text of the email.

[0015] "Importance" is an evaluation that indicates how important an email is to business operations, based on information such as the email content, sender, and subject.

[0016] "Past response history" is data that records actions and responses taken in response to previously received emails.

[0017] An "action list" is a list of specific actions to be taken next in response to received email.

[0018] "User terminal" refers to a computing device such as a computer, smartphone, or tablet used by an operator who uses the system. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention relates to a system that uses generative AI to analyze emails, automatically generate a summary of their contents, and list the next actions to be taken.

[0041] Overall system structure

[0042] This system mainly consists of a server, a terminal, and a user. The server is responsible for receiving emails, generating summaries, determining importance, and creating action lists. The terminal provides an interface for users to check the summaries and action lists and perform necessary actions.

[0043] Server Operation

[0044] The server first connects to the specified mail server and retrieves all emails received that day. These emails are then stored in a database. Generative AI is then used to analyze these emails and summarize the content of each email. The summary is then ranked based on metadata such as the sender and subject.

[0045] Next, the server predicts the next action to be taken for the generated summary based on the past response history, and creates an action list based on this prediction result.

[0046] Device behavior

[0047] The terminal displays the summary and action list sent from the server. By checking this information on the terminal, the user can quickly grasp the summary and importance of all emails received that day. In addition, the user can check the next action to be taken as shown in the action list and take specific action based on it.

[0048] How users use it

[0049] At the end of work, a user starts the system and checks the summary of the day's received emails on their device. For example, a highly important email may be a request from a superior to "check project progress," and the summary may read, "Request for a report on the progress of Project X." In this case, the action list may include a specific action such as "Prepare and send a progress report." By following this instruction, the user can quickly take the necessary action, preventing email omissions and delays in response.

[0050] In a specific implementation, for example, the system connects to a mail server using the IMAP protocol and uses Python to retrieve and analyze email content. Furthermore, a natural language processing model is used as the generative AI, and the results are used to generate summaries and predict actions. This allows users to improve work efficiency and prevent oversight of important business emails.

[0051] The processing flow will be explained below.

[0052] Step 1:

[0053] The server connects to the specified mail server using the IMAP protocol, authenticates using the user's email account information (email address and password), and selects a mailbox (usually the inbox).

[0054] Step 2:

[0055] The server searches and retrieves all emails received that day. Based on the list of email IDs obtained as a search result, it retrieves detailed information about each email and stores it in a database.

[0056] Step 3:

[0057] The server extracts the text body of each retrieved email, which includes determining whether the email is multi-part or single-part and extracting the text from the appropriate part.

[0058] Step 4:

[0059] The server inputs the extracted email text into the generation AI, which automatically generates a summary of the content. The generated summary is temporarily saved along with metadata such as the email sender and subject.

[0060] Step 5:

[0061] The server determines the importance of each email based on metadata such as the sender and subject line of the email. For example, an email from your boss is considered high importance.

[0062] Step 6:

[0063] The server combines the generated summary with the determined importance to generate summary data of the email content, which includes information about the sender, subject, summary, and importance of the email.

[0064] Step 7:

[0065] The server references past response history and uses predictive AI to predict the next action to be taken for each summary. Based on the prediction results, a specific action list is created.

[0066] Step 8:

[0067] The server sends the generated summary data and action list to the user's device, transferring data in real time via a web server or API.

[0068] Step 9:

[0069] The terminal displays the summary data and action list sent from the server, and the user can check the summary and importance of all emails and the next action to be taken through the interface on the terminal.

[0070] Step 10:

[0071] The user can then take necessary action based on the displayed summary and action list through the terminal. For example, the user can create and reply to an important inquiry from a superior by creating a progress report.

[0072] Example 1

[0073] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0074] In today's information society, email is an important means of business communication. However, because a large volume of emails are received every day, sorting and responding to them requires a lot of time and effort. In particular, important emails can be overlooked or responded to late, which can reduce business efficiency. To solve this problem, a system is needed that can efficiently analyze emails, extract necessary information, and respond quickly.

[0075] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0076] In this invention, the server includes means for analyzing emails using a generative AI and automatically generating a summary of the contents, means for determining the importance of the email, means for displaying the summary based on the determined importance, means for predicting the next action to be taken by referring to past response history and creating an action list, means for displaying the action list and summary on a user terminal, means for connecting to a mail server using the IMAP protocol and retrieving received emails, and means for inputting prompt sentences into the generative AI model and analyzing emails using natural language processing. This allows users to efficiently manage received emails, prevent overlooking important emails or delaying responses, and improve work efficiency.

[0077] "Generative AI" is an artificial intelligence model that performs natural language processing based on input data and automatically generates sentences and information.

[0078] "Email" refers to a digital letter sent and received via the Internet or other means.

[0079] A "summary" is a short summary that extracts the main information from a long piece of text or data.

[0080] "Importance" is an indicator of how much priority a particular email should have.

[0081] An "action list" is a list of specific action items to be carried out according to a specific task or instruction.

[0082] "IMAP protocol" stands for Internet Message Access Protocol, and is a standard communication protocol used when receiving email from a mail server.

[0083] A "prompt" is an instruction or question entered into a generative AI model to instruct it to perform a specific task.

[0084] "Natural language processing" refers to the technology that allows computers to understand and process human language.

[0085] A "mail server" is a server that manages and stores the sending and receiving of e-mail.

[0086] A "database" is a system for efficiently searching, storing, and updating data.

[0087] A "user terminal" is a device (e.g., a computer or smartphone) that a user directly operates to access the system.

[0088] This invention relates to a system that uses generative AI to analyze emails, automatically summarize their contents, and create a list of next actions. The system of the present invention is mainly composed of a server, a terminal, and a user.

[0089] Server Operation

[0090] The server first connects to the mail server using the IMAP protocol and retrieves all emails received that day. The software for this is the Python language and its library, imaplib. The retrieved emails are then stored in a database. SQLite can be used as the database.

[0091] The server then uses a generative AI model (e.g., a natural language processing model) to analyze these emails and summarize the content of each email. The AI ​​model receives prompts and uses natural language processing to generate summaries.

[0092] Specific examples of prompts are as follows:

[0093] "Below is the content of a new email. Please summarize it, rate its importance, and suggest a next course of action based on that."

[0094] Email body:

[0095] "Progress on Project X is delayed, so we request an immediate response. Details will be explained at the meeting."

[0096] The generated summary is then ranked based on metadata such as the sender and subject line, and the server then uses past response history to predict the next course of action, creating an action list based on that.

[0097] Device behavior

[0098] The terminal displays the summary and action list sent from the server. A web browser is generally used as the user interface. JavaScript (registered trademark) and HTML are used to enable the user to quickly grasp the summary and importance of all received emails.

[0099] How users use it

[0100] At the end of work, the user starts the system and checks the summary of the day's received emails on their device. For example, if a highly important email is a request for "project progress confirmation" and the summary reads "Requesting a report on the progress of Project X," the action list will include a specific action such as "Prepare and send a progress report." Users can follow these instructions to quickly take the necessary action, preventing emails from being overlooked or delaying responses.

[0101] In this way, the system of the present invention effectively analyzes emails using generative AI and provides users with useful summaries and specific action lists, thereby significantly improving the efficiency of their business management.

[0102] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0103] Step 1:

[0104] The server connects to the mail server using the IMAP protocol and retrieves all emails received that day. Specifically, it connects using the Python imaplib library and retrieves emails using the search method.

[0105] Input: Mail server account information (ID, password)

[0106] Output: Received email data

[0107] Specific behavior: The server establishes a connection to the mail server and retrieves all emails from the specified folder (e.g., inbox).

[0108] Step 2:

[0109] The server stores the retrieved emails in a database, which uses SQLite.

[0110] Input: Received email data

[0111] Output: Email data stored in a database (sender, subject, body, etc.)

[0112] Specific operation: The server analyzes the email data it receives, extracts information such as the sender, subject, and body of the email, and stores it in an SQLite database.

[0113] Step 3:

[0114] The server uses a generative AI model to analyze the stored emails and summarize the content of each one.

[0115] Input: Email body in database

[0116] Output: Generated summary

[0117] Specific operation: The server generates a prompt sentence for the AI ​​model, uses it as input for natural language processing, and generates a summary sentence.

[0118] Step 4:

[0119] The server determines the importance based on the content of the summarized summary.

[0120] Input: Generated summary

[0121] Output: Importance (high, low, etc.)

[0122] Specific operation: The server determines the importance by branching conditions based on keywords in the summary (e.g., "important," "urgent," etc.).

[0123] Step 5:

[0124] The server refers to the past response history, predicts the next action to be taken for the generated summary, and creates an action list.

[0125] Input: Generated summary, past correspondence history

[0126] Output: Action list

[0127] Specific operation: The server refers to a database of past cases and, based on similar past cases and response history, inputs prompts into the AI ​​model to predict the next action to be taken, generating the results as an action list.

[0128] Step 6:

[0129] The terminal displays a summary and an action list.

[0130] Input: Summary, Action List

[0131] Output: Summary and action list displayed in the user interface

[0132] Specific operation: The terminal displays the summary and action list through an interface such as a web browser, allowing the user to check them.

[0133] Step 7:

[0134] The user checks the summary and action list displayed on the terminal and takes the necessary action.

[0135] Input: Displayed summary and action list

[0136] Output: Actions taken (creating and sending progress reports, etc.)

[0137] Specific actions: The user operates the device to check the summary and action list and take action according to the instructions. For example, a specific action such as creating and sending a progress report is performed.

[0138] Through these specific processing steps, the system can efficiently manage received emails and prevent important emails from being overlooked or delayed in response.

[0139] (Application example 1)

[0140] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0141] With current electronic communications analysis systems, classifying and responding to a vast amount of communication usage requires time and effort, making it difficult to respond quickly and efficiently. Particularly in customer service operations, there is a risk of overlooking important communication usage, which can lead to delays in appropriate responses. It is also difficult to predict the next appropriate action to take based on response history, leading to lower customer satisfaction and a deterioration in work efficiency.

[0142] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0143] In this invention, the server includes means for analyzing electronic communications using a generation AI and automatically generating a summary of the content, means for determining the importance of the electronic communications, means for displaying the summary based on the determined importance, means for predicting the next action to be taken by referring to past response history and creating a response list, means for displaying the response list and summary on a user terminal, means for acquiring new electronic communications, means for automatically generating the next action to be taken from the summarized content and supporting customer responses, and means for summarizing and predicting actions using natural language processing with the generation AI. This makes it possible to quickly and efficiently identify important communication uses from a huge amount of electronic communications and provide appropriate customer responses.

[0144] "Generative AI" refers to technologies and systems that use artificial intelligence to generate data and information.

[0145] "Electronic communications" are digital forms of communication sent or received through email, messaging apps, etc.

[0146] A "summary" is a brief description of the contents of an electronic communication that abridges the content and extracts only the main points.

[0147] "Importance" is an indicator of how important the content of electronic communication is to business operations or responses.

[0148] "Target history" is a record of the responses and processes that have been carried out up to now.

[0149] An "action list" is a list of specific next steps to be taken based on the content of an electronic communication.

[0150] A "user terminal" is a device such as a computer or smart device used by a user.

[0151] "Natural language processing" is a technology that allows computers to understand natural language, and is a method for analyzing and generating text.

[0152] "Customer service" refers to the work that companies and organizations do to respond to inquiries and requests from customers.

[0153] This invention relates to a system that uses generative AI to analyze electronic communications, automatically generate a summary of the content, and list the next actions to be taken. The overall structure of this system mainly consists of a server, a terminal, and a user.

[0154] Server Operation

[0155] The server first connects to the specified mail server and retrieves new electronic communications. The retrieved electronic communications are stored in a database. Then, a generative AI is used to analyze these electronic communications and summarize the content of each communication. A natural language processing model is used as this generative AI. For example, the OpenAI (registered trademark) API can be used.

[0156] The importance of the summarized communication content is determined based on metadata such as the sender and subject. Next, the server predicts the next action to be taken for the generated summary based on past response history, and creates a response list based on this prediction result.

[0157] Hardware and software used

[0158] Hardware: Servers, user devices (smartphones, PCs, etc.)

[0159] Software: Python, IMAP protocol, OpenAI API

[0160] Device behavior

[0161] The terminal displays the summary and action list sent from the server. The user can check this information on the terminal and quickly grasp the summary and importance of all electronic communications received that day. The user can also check the next action indicated in the action list and take specific action based on it. For example, this can be done using an application installed on a smartphone or head-mounted display.

[0162] How users use it

[0163] At the end of the workday, the user starts up the system and checks a summary of the day's incoming communications on their terminal. For example, if a high-priority communication is a customer inquiry about being unable to make a payment, and the summary reads, "We have received an inquiry about an issue preventing payment," the response list will include a specific action: "Send troubleshooting instructions for the payment system." Following these instructions, the user can quickly take the necessary action, preventing oversights and delays in customer support.

[0164] Prompt Sentence Examples

[0165] Summary generation prompt:

[0166] "Please summarize the email below:\n\nI'm inquiring about an inability to make a payment. What are the troubleshooting steps?"

[0167] Matching list generating prompt statement:

[0168] "Based on this summary, please list the next steps you should take:\n\nWe've received a call about an issue with being unable to make a payment."

[0169] This will realize a system that can quickly and efficiently identify important communications from among a vast amount of electronic communications and provide appropriate customer support.

[0170] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0171] Step 1:

[0172] The server connects to the designated mail server to retrieve new electronic communications.

[0173] Input: Mail server connection information and authentication information

[0174] Output: Newly received electronic communications

[0175] What it does: It uses the IMAP protocol to connect to a mail server, retrieve new electronic communications, and store them in a database.

[0176] Step 2:

[0177] The server uses a generation AI to analyze the acquired electronic communications and automatically generate a summary of the contents.

[0178] Input: Retrieved electronic communication

[0179] Output: Generated summary

[0180] Specific operation: Uses the OpenAI API to analyze the content of acquired electronic communications and generate a summary. For example, input the body of an email as a prompt and receive the summary result.

[0181] Step 3:

[0182] The server determines the importance of the generated summary.

[0183] Input: Generated summary and email metadata (sender, subject, etc.)

[0184] Output: Importance judgment result

[0185] Specific operation: An algorithm is run to determine the importance of an email based on the summary and the email's metadata, and past importance determination data is referenced.

[0186] Step 4:

[0187] The server refers to past response history, predicts the next action to be taken, and creates a response list.

[0188] Input: Generated summary and past correspondence history

[0189] Output: Correspondence list

[0190] Specific operation: Using generative AI, analyze the summary content and past response history, predict the appropriate next action to take, and generate a response list. Instructions are given to the AI ​​using prompt sentences.

[0191] Step 5:

[0192] The server transmits the correspondence list and the summary to the user terminal for display.

[0193] Input: Correspondence list and summary

[0194] Output: Corresponding list and summary displayed on user terminal

[0195] Specific operation: The server sends a correspondence list and summary to the user's smartphone or PC, and displays it using a dedicated application.

[0196] Step 6:

[0197] The user checks the summary and correspondence list displayed on the terminal and then performs the next action to be taken.

[0198] Input: Summary and corresponding list displayed on terminal

[0199] Output: Specific actions taken

[0200] Specific actions: The user checks the summary and correspondence list on the terminal and promptly takes necessary actions based on it (e.g., submitting troubleshooting instructions for the payment system).

[0201] The above processing steps realize a system that can quickly and efficiently identify important communications from among a vast amount of electronic communications and provide appropriate customer support.

[0202] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0203] This invention relates to a system that uses generative AI to analyze emails and automatically generate summaries of their contents, as well as a system that incorporates an emotion engine that recognizes the user's emotions, allowing the reevaluation of email importance and customization of action lists.

[0204] Overall system configuration

[0205] The system consists of the following components:

[0206] 1. Server: Receives emails, analyzes their contents, generates summaries, determines their importance, and creates action lists.

[0207] 2. User terminal: displays a summary and action list and monitors the user's emotional state.

[0208] 3. User: Uses the system to check and respond to emails.

[0209] Server Operation

[0210] The server connects to the specified mail server, retrieves, analyzes, summarizes, and determines the importance of emails based on their content.The server then refers to past response history and uses predictive AI to predict the next action to be taken, generating an action list.

[0211] Furthermore, the system uses an emotion engine to analyze the user's emotional state and reassess the importance of the email based on that information. Based on this reassessment, the system customizes the action list and prioritizes suggested responses according to the user's emotional state.

[0212] User terminal operation

[0213] The user device displays the summary and action list sent from the server in real time. The device is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice via a camera and microphone to recognize their emotional state.

[0214] This emotional state is monitored in real time and fed back to the server, which then dynamically adjusts the display format and content of the summary and action list.

[0215] How users use it

[0216] The user starts the system at the end of work and checks the summary of the day's received emails on the device. For example, an email from a boss asking for a project progress report is displayed in the summary as being of high importance. If the user's emotional state indicates "stress," the system will prioritize actions that help reduce stress, such as "relax before starting to write a progress report," in the action list.

[0217] Specifically, while a user is checking their email on the screen, the device's camera analyzes their facial expressions and the microphone analyzes their tone of voice to recognize their emotional state. Meanwhile, the server receives this data in real time and dynamically adjusts the email display and action list. By receiving this interactive feedback, users can respond to their work efficiently while taking their emotions into consideration.

[0218] Specifically, when a user is feeling stressed or anxious, the emotion engine can automatically prioritize email processing taking their emotions into consideration. It can also take actions or make suggestions to encourage relaxation as needed, reducing the user's mental burden.

[0219] In this way, by combining generative AI and an emotion engine, the present invention provides a novel system that improves the efficiency of email processing and subsequent actions while taking into account the user's emotional state.

[0220] The processing flow will be explained below.

[0221] Step 1:

[0222] The server connects to the mail server using the IMAP protocol and retrieves the emails received that day from the specified mailbox (usually the inbox), which are then stored in a database.

[0223] Step 2:

[0224] The server extracts the text body of each retrieved email, which includes determining whether the email is multi-part or single-part and extracting the text from the appropriate part.

[0225] Step 3:

[0226] The server then uses generative AI to analyze the extracted email text and automatically generate a summary of the email's contents, which concisely summarizes the main points of the email.

[0227] Step 4:

[0228] The server determines the importance of an email based on metadata such as the sender and subject line of the email. For example, an email from your boss or about an important project might be considered high importance.

[0229] Step 5:

[0230] The server combines the generated summary with the determined importance to generate summary data, which includes information about the sender, subject, summary, and importance of the email.

[0231] Step 6:

[0232] The device uses a camera and microphone to analyze the user's facial expressions and tone of voice in real time to recognize the user's emotional state, thereby determining the user's stress level and emotional state.

[0233] Step 7:

[0234] The server receives the user's emotional state data and re-evaluates the importance of the emails based on the data, and determines the order in which the emails are processed that is optimal for the user's current emotional state.

[0235] Step 8:

[0236] The server references past response history and uses predictive AI to predict the next action to be taken for the generated summary. Based on this prediction, a specific action list is created.

[0237] Step 9:

[0238] The server generates a customized action list taking into account the user's emotional state and sends it to the user's device along with summary data. Based on the user's emotional state, actions that help reduce stress and urgent emails are prioritized.

[0239] Step 10:

[0240] The terminal displays the summary data sent from the server and a customized action list, through which the user can check a summary of all emails, their importance, and the next action to be taken.

[0241] Step 11:

[0242] The user can take necessary actions based on the summary and action list displayed on the device, such as sending a progress report in response to an important email from a boss, or performing a relaxation action when stress levels are high.

[0243] Through this series of processes, the system takes into account the user's emotional state to improve the efficiency of email processing and subsequent actions, thereby reducing the user's mental burden.

[0244] Example 2

[0245] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0246] In today's work environment, there is a huge amount of electronic communication, and people need to be able to quickly extract important information from it and respond appropriately. However, this takes time and effort, and it is especially difficult to respond effectively when users are stressed or tired. Furthermore, many current systems do not take users' emotional state into account and can only provide fixed responses, which hinders users' efficient work performance.

[0247] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for analyzing electronic communications using a generation AI and automatically generating a summary of the contents, means for determining the importance of the electronic communications, means for displaying the summary based on the determined importance, means for predicting the next action to be taken by referring to past response history and creating a work instruction list, means for displaying the work instruction list and summary on a user terminal, means for monitoring the user's emotional state, reevaluating the importance based on that information, and customizing the work instruction list, and means for analyzing emotional data in real time and feeding it back to the server. This enables efficient information extraction and response that takes the user's emotional state into consideration.

[0248] "Generative AI" is a technology that uses artificial intelligence to generate and analyze data.

[0249] "Electronic communications" means digital information sent or received over the Internet or an intranet.

[0250] A summary is a concise summary of the important points of a longer piece of text or data.

[0251] "Importance" is a measure of the priority or importance associated with particular information or data.

[0252] "Past response history" refers to records of responses and processes that have been taken in the past.

[0253] A "work instruction list" is a list of actions and procedures required to carry out a specific job or task.

[0254] A "user terminal" is a device that a user directly operates to display and input information. Generally, this includes PCs and smartphones.

[0255] "Emotional state" is a term that refers to a user's psychological state or mood.

[0256] "Feedback" refers to information or data given regarding the behavior or results of a system.

[0257] This invention relates to a system that uses generative AI to analyze electronic communications and automatically generate summaries of their contents, and also incorporates an emotion engine that recognizes the user's emotions, enabling the reevaluation of the importance of communications and the customization of action instruction lists.

[0258] Server Operation

[0259] The server first connects to the designated mail server to receive the electronic communication, which is accomplished using standard protocols such as IMAP (Internet Message Access Protocol) or POP3 (Post Office Protocol 3).

[0260] Examples:

[0261] The server connects to a communication server using the IMAP protocol and obtains new communications.

[0262] The captured communications are stored in an internal database.

[0263] The server then uses a generative AI model (such as OpenAI's GPT-4®) to summarize the content of the received communication, designing an appropriate prompt and passing it to the generative AI.

[0264] Example prompt sentence:

[0265] "Please summarize the contents of the following electronic communication: 'Details about the progress of the project...'"

[0266] Based on the generated summary, the server uses an algorithm to determine the importance of the communication, based on factors such as past response history and the frequency of occurrence of fixed keywords.

[0267] Furthermore, the server uses the prediction function of the generative AI to predict the next action to be taken and create a list of action instructions. For example, for a communication about "checking project progress," which is a high-priority item, it suggests a specific action such as "submit a progress report."

[0268] Device behavior

[0269] The user device displays the summary and action instruction list sent from the server in real time, and the device's built-in emotion engine analyzes the user's facial expressions and tone of voice via a camera and microphone to recognize their emotional state.

[0270] Examples:

[0271] While the user is checking the communication, the device's camera captures the user's facial expressions, and the emotion engine performs emotion analysis in real time.

[0272] The microphone analyzes the tone of the user's voice and extracts data on the level of stress and excitement.

[0273] The extracted emotion data is fed back to the server, which then uses this data to reevaluate the importance of the communication and the action instruction list, customizing them as necessary.

[0274] How users use it

[0275] The user starts the system at the end of work and checks a summary of the day's received communications on their device. For example, a communication from a superior about "checking project progress" is displayed in the summary as being of high importance. If the user's emotional state indicates "stress," the system will prioritize actions that help reduce stress, such as "relax before starting to write a progress report," in the list of action instructions.

[0276] Hardware and software used

[0277] The following hardware and software are used to implement this system:

[0278] Server: A server machine that handles communication (e.g., a Linux server)

[0279] Generative AI models: such as OpenAI's GPT-4

[0280] Emotion engine: Software with the ability to analyze the user's facial expressions and voice (e.g., Microsoft® Azure® Face API and Speech API)

[0281] User device: PC or smartphone equipped with a camera and microphone

[0282] Through the above steps, the present invention realizes a series of processes that analyzes electronic communications, generates summaries, and provides action instructions that take emotional states into account. By using this system, users can process information and respond to business tasks efficiently while taking their emotions into consideration.

[0283] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0284] Step 1:

[0285] The server receives new electronic communications from a designated mail server.

[0286] Specific behavior:

[0287] The server connects to the mail server using the IMAP or POP3 protocol.

[0288] Obtain a list of new electronic communications and store their details in an internal database.

[0289] Input: mail server connection information, query for new electronic communications.

[0290] Output: A list of retrieved new electronic communications.

[0291] Step 2:

[0292] The server uses the generative AI model to summarize the content of the received electronic communication.

[0293] Specific behavior:

[0294] The server passes the following prompt to the generative AI model:

[0295] "Please summarize the contents of the following electronic communication: 'Details about the progress of the project...'"

[0296] The generative AI model returns a summary and stores the results in an internal database.

[0297] Input: Content of received electronic communication, prompt text.

[0298] Output: The generated summary.

[0299] Step 3:

[0300] The server determines the importance of the email.

[0301] Specific behavior:

[0302] The server uses an importance determination algorithm to rate the importance of the received electronic communication.

[0303] The importance score is stored in an internal database.

[0304] Input: Received electronic communications, summaries, and past correspondence history.

[0305] Output: Importance score.

[0306] Step 4:

[0307] The server predicts what action to take next and creates an action list.

[0308] Specific behavior:

[0309] The server passes the following prompt to the generative AI model:

[0310] "Suggest next actions for high priority emails"

[0311] The generative AI model suggests specific actions and stores the results as an action list in an internal database.

[0312] Input: Importance score, prompt for the generation AI.

[0313] Output: Action list.

[0314] Step 5:

[0315] The device monitors the user's emotional state and provides feedback to the server.

[0316] Specific behavior:

[0317] The camera captures the user's facial expressions, and the emotion engine analyzes them.

[0318] The microphone analyzes the tone of the user's voice and extracts emotional data.

[0319] The emotion data is sent to the server.

[0320] Input: User's facial expression data, tone of voice data.

[0321] Output: Emotion data.

[0322] Step 6:

[0323] The server reevaluates and customizes the importance and action list of emails based on emotional data.

[0324] Specific behavior:

[0325] The server receives the emotion data and evaluates the importance and action list again.

[0326] If necessary, add customizations to the action list, such as "Relax and then continue working."

[0327] Save the updated action list to an internal database.

[0328] Input: sentiment data, original action list, importance score.

[0329] Output: The updated action list.

[0330] Step 7:

[0331] The terminal displays a summary and a customized action list to the user.

[0332] Specific behavior:

[0333] The device retrieves the summary and latest action list from the server.

[0334] Display a summary and action list on the user interface.

[0335] Input: Summary, action list from server.

[0336] Output: The summary and action list that is displayed to the user.

[0337] The above is the specific processing flow of this system. Each processing step works together to realize efficient analysis and response of electronic communications taking into account the user's emotional state.

[0338] (Application example 2)

[0339] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0340] In conventional electronic payment services, users had to manually check and respond to all notifications and transaction emails, which was cumbersome and often mentally taxing. Furthermore, because the system did not take into account the user's emotional state, there were issues such as missing important transactions in stressful situations or being unable to make appropriate decisions regarding high-risk transactions.

[0341] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing emails using a generation AI and automatically generating a summary of the contents, means for determining the importance of the email, means for displaying the summary based on the determined importance, means for predicting the next action to be taken by referring to past response history and creating an action list, means for displaying the action list and the summary on a user terminal, means for analyzing the user's emotional state using a camera and microphone, and means for reevaluating the importance based on the user's emotional state and customizing the action list. This enables the user to make optimal decisions taking into consideration their emotional state and efficiently process important notifications and transactions related to electronic payments.

[0342] "Generative AI" refers to technology that uses artificial intelligence models to automatically generate text and data.

[0343] "Email" is a means of communication for sending and receiving text information, files, etc. via the Internet.

[0344] The "summary" is information that briefly summarizes the contents of the email.

[0345] "Importance" is a metric used to assess the urgency and priority of an email or notification.

[0346] "Past response history" refers to a record of responses to emails that a user has made up to now.

[0347] An "action list" is a list of tasks that indicate the next action the user should take.

[0348] "User terminal" is a general term for electronic devices used by users, including smartphones and tablets.

[0349] A "camera" is a device that captures video or still images.

[0350] A "microphone" is a device that records sound.

[0351] An "emotional state" is an emotion a user is feeling at a particular point in time, and may include "stressed" or "relaxed."

[0352] "Analysis" is a method of examining data and information in detail and understanding its content.

[0353] "Reevaluation" means to repeat or review an evaluation that has already been made.

[0354] "Customization" refers to adjusting specific content or settings to suit the user's needs and circumstances.

[0355] This invention is a system that uses generative AI to analyze emails and automatically generate summaries of their contents, and also incorporates an emotion engine that recognizes the user's emotions, enabling the re-evaluation of email importance and customization of action lists. This system consists of the following components:

[0356] Server Operation

[0357] The server connects to the specified mail server, retrieves, analyzes, and summarizes emails, and determines their importance based on their content. It also references past response history to predict the next action to be taken and uses generative AI to create an action list. The server analyzes the user's emotional state through a camera or microphone, and based on that information, reevaluates the importance of emails and customizes the action list. Examples of generative AI models used include GPT-3 (registered trademark) and BERT.

[0358] User terminal operation

[0359] The user's device displays the summary and action list sent from the server in real time. The device is equipped with an emotion engine that uses a camera and microphone to analyze the user's facial expressions and tone of voice to recognize their emotional state. This emotional state is fed back to the server in real time, enabling the device to respond in a way that takes the user's emotions into consideration.

[0360] Specific examples

[0361] For example, when a user launches the application at the end of work and checks the summary of the day's received emails on their device, an email from their boss asking them to "check the project progress" is displayed in the summary as being of high importance. If the user's emotional state indicates "stress," the system will prioritize actions that help reduce stress, such as "relax before starting to write a progress report," in the action list.

[0362] Hardware and software used

[0363] Camera: Used to capture the user's facial expressions.

[0364] Microphone: Used to analyze the tone of the user's voice.

[0365] Python standard library: used to retrieve and parse emails (smtplib, imaplib, email).

[0366] TextBlob: Used to summarize the email content.

[0367] Emotion Recognition: A library for analyzing user emotions.

[0368] Prompt Sentence Examples

[0369] Summarize emails about "payment notifications" and "important transaction confirmations," determine their importance based on the user's emotional state, and generate appropriate action suggestions. If the user's emotion is determined to be "stressed," suggest that they relax before proceeding.

[0370] This allows users to make optimal decisions while taking their emotions into consideration, and efficiently take appropriate actions regarding electronic payments and mail processing.

[0371] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0372] Step 1:

[0373] The server connects to the specified mail server and retrieves emails. It requires mail server login information as input and provides the retrieved email data as output.

[0374] Step 2:

[0375] The server analyzes the retrieved email and summarizes its contents. The email body is required as input, and a generative AI model (e.g., TextBlob) is used to generate a summary, resulting in a summary text as output.

[0376] Step 3:

[0377] The server determines the importance of an email. It takes parsed email metadata (sender, subject, etc.) and summary text as input, evaluates the importance using an importance determination algorithm, and outputs an importance score.

[0378] Step 4:

[0379] The server references past correspondence history and predicts the next action to be taken. Emails and past correspondence history data are required as input, and a generative AI model is used to make the prediction, resulting in a list of actions as output.

[0380] Step 5:

[0381] The server transmits the action list and summary to the user terminal. The generated action list and summary text are required as input, and data transmission to the user terminal is required as output.

[0382] Step 6:

[0383] The terminal displays the summary and action list sent from the server in real time. It requires data from the server as input and provides visual information to the user as output.

[0384] Step 7:

[0385] The device analyzes the user's emotional state using a camera and microphone. This process requires the user's facial expression and voice data as input, analyzes the emotional state using an emotion recognition algorithm, and obtains emotional data as output.

[0386] Step 8:

[0387] The server receives the user's emotional data, re-evaluates the importance of emails based on the data, and customizes the action list. The server requires the emotional data and the existing action list as input, adjusts the action list using a re-evaluation algorithm, and obtains the customized action list as output.

[0388] Step 9:

[0389] Again, the server sends the customized action list to the user terminal and suggests appropriate actions to the user. A customized action list is required as input, and the final action list is provided to the user as output.

[0390] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0391] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0392] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0393] [Second embodiment]

[0394] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0395] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0396] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0397] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0398] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0399] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0400] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0401] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0402] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0403] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0404] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0405] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0406] This invention relates to a system that uses generative AI to analyze emails, automatically generate a summary of their contents, and list the next actions to be taken.

[0407] Overall system structure

[0408] This system mainly consists of a server, a terminal, and a user. The server is responsible for receiving emails, generating summaries, determining importance, and creating action lists. The terminal provides an interface for users to check the summaries and action lists and perform necessary actions.

[0409] Server Operation

[0410] The server first connects to the specified mail server and retrieves all emails received that day. These emails are then stored in a database. Generative AI is then used to analyze these emails and summarize the content of each email. The summary is then ranked based on metadata such as the sender and subject.

[0411] Next, the server predicts the next action to be taken for the generated summary based on the past response history, and creates an action list based on this prediction result.

[0412] Device behavior

[0413] The terminal displays the summary and action list sent from the server. By checking this information on the terminal, the user can quickly grasp the summary and importance of all emails received that day. In addition, the user can check the next action to be taken as shown in the action list and take specific action based on it.

[0414] How users use it

[0415] At the end of work, a user starts the system and checks the summary of the day's received emails on their device. For example, a highly important email may be a request from a superior to "check project progress," and the summary may read, "Request for a report on the progress of Project X." In this case, the action list may include a specific action such as "Prepare and send a progress report." By following this instruction, the user can quickly take the necessary action, preventing email omissions and delays in response.

[0416] In a specific implementation, for example, the system connects to a mail server using the IMAP protocol and uses Python to retrieve and analyze email content. Furthermore, a natural language processing model is used as the generative AI, and the results are used to generate summaries and predict actions. This allows users to improve work efficiency and prevent oversight of important business emails.

[0417] The processing flow will be explained below.

[0418] Step 1:

[0419] The server connects to the specified mail server using the IMAP protocol, authenticates using the user's email account information (email address and password), and selects a mailbox (usually the inbox).

[0420] Step 2:

[0421] The server searches and retrieves all emails received that day. Based on the list of email IDs obtained as a search result, it retrieves detailed information about each email and stores it in a database.

[0422] Step 3:

[0423] The server extracts the text body of each retrieved email, which includes determining whether the email is multi-part or single-part and extracting the text from the appropriate part.

[0424] Step 4:

[0425] The server inputs the extracted email text into the generation AI, which automatically generates a summary of the content. The generated summary is temporarily saved along with metadata such as the email sender and subject.

[0426] Step 5:

[0427] The server determines the importance of each email based on metadata such as the sender and subject line of the email. For example, an email from your boss is considered high importance.

[0428] Step 6:

[0429] The server combines the generated summary with the determined importance to generate summary data of the email content, which includes information about the sender, subject, summary, and importance of the email.

[0430] Step 7:

[0431] The server references past response history and uses predictive AI to predict the next action to be taken for each summary. Based on the prediction results, a specific action list is created.

[0432] Step 8:

[0433] The server sends the generated summary data and action list to the user's device, transferring data in real time via a web server or API.

[0434] Step 9:

[0435] The terminal displays the summary data and action list sent from the server, and the user can check the summary and importance of all emails and the next action to be taken through the interface on the terminal.

[0436] Step 10:

[0437] The user can then take necessary action based on the displayed summary and action list through the terminal. For example, the user can create and reply to an important inquiry from a superior by creating a progress report.

[0438] Example 1

[0439] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0440] In today's information society, email is an important means of business communication. However, because a large volume of emails are received every day, sorting and responding to them requires a lot of time and effort. In particular, important emails can be overlooked or responded to late, which can reduce business efficiency. To solve this problem, a system is needed that can efficiently analyze emails, extract necessary information, and respond quickly.

[0441] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0442] In this invention, the server includes means for analyzing emails using a generative AI and automatically generating a summary of the contents, means for determining the importance of the email, means for displaying the summary based on the determined importance, means for predicting the next action to be taken by referring to past response history and creating an action list, means for displaying the action list and summary on a user terminal, means for connecting to a mail server using the IMAP protocol and retrieving received emails, and means for inputting prompt sentences into the generative AI model and analyzing emails using natural language processing. This allows users to efficiently manage received emails, prevent overlooking important emails or delaying responses, and improve work efficiency.

[0443] "Generative AI" is an artificial intelligence model that performs natural language processing based on input data and automatically generates sentences and information.

[0444] "Email" refers to a digital letter sent and received via the Internet or other means.

[0445] A "summary" is a short summary that extracts the main information from a long piece of text or data.

[0446] "Importance" is an indicator of how much priority a particular email should have.

[0447] An "action list" is a list of specific action items to be carried out according to a specific task or instruction.

[0448] "IMAP protocol" stands for Internet Message Access Protocol, and is a standard communication protocol used when receiving email from a mail server.

[0449] A "prompt" is an instruction or question entered into a generative AI model to instruct it to perform a specific task.

[0450] "Natural language processing" refers to the technology that allows computers to understand and process human language.

[0451] A "mail server" is a server that manages and stores the sending and receiving of e-mail.

[0452] A "database" is a system for efficiently searching, storing, and updating data.

[0453] A "user terminal" is a device (e.g., a computer or smartphone) that a user directly operates to access the system.

[0454] This invention relates to a system that uses generative AI to analyze emails, automatically summarize their contents, and create a list of next actions. The system of the present invention is mainly composed of a server, a terminal, and a user.

[0455] Server Operation

[0456] The server first connects to the mail server using the IMAP protocol and retrieves all emails received that day. The software for this is the Python language and its library, imaplib. The retrieved emails are then stored in a database. SQLite can be used as the database.

[0457] The server then uses a generative AI model (e.g., a natural language processing model) to analyze these emails and summarize the content of each email. The AI ​​model receives prompts and uses natural language processing to generate summaries.

[0458] Specific examples of prompts are as follows:

[0459] "Below is the content of a new email. Please summarize it, rate its importance, and suggest a next course of action based on that."

[0460] Email body:

[0461] "Progress on Project X is delayed, so we request an immediate response. Details will be explained at the meeting."

[0462] The generated summary is then ranked based on metadata such as the sender and subject line, and the server then uses past response history to predict the next course of action, creating an action list based on that.

[0463] Device behavior

[0464] The terminal displays the summary and action list sent from the server. A web browser is generally used as the user interface. JavaScript and HTML are used to enable the user to quickly grasp the summary and importance of all received emails.

[0465] How users use it

[0466] At the end of work, the user starts the system and checks the summary of the day's received emails on their device. For example, if a highly important email is a request for "project progress confirmation" and the summary reads "Requesting a report on the progress of Project X," the action list will include a specific action such as "Prepare and send a progress report." Users can follow these instructions to quickly take the necessary action, preventing emails from being overlooked or delaying responses.

[0467] In this way, the system of the present invention effectively analyzes emails using generative AI and provides users with useful summaries and specific action lists, thereby significantly improving the efficiency of their business management.

[0468] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0469] Step 1:

[0470] The server connects to the mail server using the IMAP protocol and retrieves all emails received that day. Specifically, it connects using the Python imaplib library and retrieves emails using the search method.

[0471] Input: Mail server account information (ID, password)

[0472] Output: Received email data

[0473] Specific behavior: The server establishes a connection to the mail server and retrieves all emails from the specified folder (e.g., inbox).

[0474] Step 2:

[0475] The server stores the retrieved emails in a database, which uses SQLite.

[0476] Input: Received email data

[0477] Output: Email data stored in a database (sender, subject, body, etc.)

[0478] Specific operation: The server analyzes the email data it receives, extracts information such as the sender, subject, and body of the email, and stores it in an SQLite database.

[0479] Step 3:

[0480] The server uses a generative AI model to analyze the stored emails and summarize the content of each one.

[0481] Input: Email body in database

[0482] Output: Generated summary

[0483] Specific operation: The server generates a prompt sentence for the AI ​​model, uses it as input for natural language processing, and generates a summary sentence.

[0484] Step 4:

[0485] The server determines the importance based on the content of the summarized summary.

[0486] Input: Generated summary

[0487] Output: Importance (high, low, etc.)

[0488] Specific operation: The server determines the importance by branching conditions based on keywords in the summary (e.g., "important," "urgent," etc.).

[0489] Step 5:

[0490] The server refers to the past response history, predicts the next action to be taken for the generated summary, and creates an action list.

[0491] Input: Generated summary, past correspondence history

[0492] Output: Action list

[0493] Specific operation: The server refers to a database of past cases and, based on similar past cases and response history, inputs prompts into the AI ​​model to predict the next action to be taken, generating the results as an action list.

[0494] Step 6:

[0495] The terminal displays a summary and an action list.

[0496] Input: Summary, Action List

[0497] Output: Summary and action list displayed in the user interface

[0498] Specific operation: The terminal displays the summary and action list through an interface such as a web browser, allowing the user to check them.

[0499] Step 7:

[0500] The user checks the summary and action list displayed on the terminal and takes the necessary action.

[0501] Input: Displayed summary and action list

[0502] Output: Actions taken (creating and sending progress reports, etc.)

[0503] Specific actions: The user operates the device to check the summary and action list and take action according to the instructions. For example, a specific action such as creating and sending a progress report is performed.

[0504] Through these specific processing steps, the system can efficiently manage received emails and prevent important emails from being overlooked or delayed in response.

[0505] (Application example 1)

[0506] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0507] With current electronic communications analysis systems, classifying and responding to a vast amount of communication usage requires time and effort, making it difficult to respond quickly and efficiently. Particularly in customer service operations, there is a risk of overlooking important communication usage, which can lead to delays in appropriate responses. It is also difficult to predict the next appropriate action to take based on response history, leading to lower customer satisfaction and a deterioration in work efficiency.

[0508] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0509] In this invention, the server includes means for analyzing electronic communications using a generation AI and automatically generating a summary of the content, means for determining the importance of the electronic communications, means for displaying the summary based on the determined importance, means for predicting the next action to be taken by referring to past response history and creating a response list, means for displaying the response list and summary on a user terminal, means for acquiring new electronic communications, means for automatically generating the next action to be taken from the summarized content and supporting customer responses, and means for summarizing and predicting actions using natural language processing with the generation AI. This makes it possible to quickly and efficiently identify important communication uses from a huge amount of electronic communications and provide appropriate customer responses.

[0510] "Generative AI" refers to technologies and systems that use artificial intelligence to generate data and information.

[0511] "Electronic communications" are digital forms of communication sent or received through email, messaging apps, etc.

[0512] A "summary" is a brief description of the contents of an electronic communication that abridges the content and extracts only the main points.

[0513] "Importance" is an indicator of how important the content of electronic communication is to business operations or responses.

[0514] "Target history" is a record of the responses and processes that have been carried out up to now.

[0515] An "action list" is a list of specific next steps to be taken based on the content of an electronic communication.

[0516] A "user terminal" is a device such as a computer or smart device used by a user.

[0517] "Natural language processing" is a technology that allows computers to understand natural language, and is a method for analyzing and generating text.

[0518] "Customer service" refers to the work that companies and organizations do to respond to inquiries and requests from customers.

[0519] This invention relates to a system that uses generative AI to analyze electronic communications, automatically generate a summary of the content, and list the next actions to be taken. The overall structure of this system mainly consists of a server, a terminal, and a user.

[0520] Server Operation

[0521] The server first connects to the specified mail server and retrieves new electronic communications. The retrieved electronic communications are stored in a database. Then, a generative AI is used to analyze these electronic communications and summarize the content of each communication. A natural language processing model is used as the generative AI. For example, OpenAI's API can be used.

[0522] The importance of the summarized communication content is determined based on metadata such as the sender and subject. Next, the server predicts the next action to be taken for the generated summary based on past response history, and creates a response list based on this prediction result.

[0523] Hardware and software used

[0524] Hardware: Servers, user devices (smartphones, PCs, etc.)

[0525] Software: Python, IMAP protocol, OpenAI API

[0526] Device behavior

[0527] The terminal displays the summary and action list sent from the server. The user can check this information on the terminal and quickly grasp the summary and importance of all electronic communications received that day. The user can also check the next action indicated in the action list and take specific action based on it. For example, this can be done using an application installed on a smartphone or head-mounted display.

[0528] How users use it

[0529] At the end of the workday, the user starts up the system and checks a summary of the day's incoming communications on their terminal. For example, if a high-priority communication is a customer inquiry about being unable to make a payment, and the summary reads, "We have received an inquiry about an issue preventing payment," the response list will include a specific action: "Send troubleshooting instructions for the payment system." Following these instructions, the user can quickly take the necessary action, preventing oversights and delays in customer support.

[0530] Prompt Sentence Examples

[0531] Summary generation prompt:

[0532] "Please summarize the email below:\n\nI'm inquiring about an inability to make a payment. What are the troubleshooting steps?"

[0533] Matching list generating prompt statement:

[0534] "Based on this summary, please list the next steps you should take:\n\nWe've received a call about an issue with being unable to make a payment."

[0535] This will realize a system that can quickly and efficiently identify important communications from among a vast amount of electronic communications and provide appropriate customer support.

[0536] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0537] Step 1:

[0538] The server connects to the designated mail server to retrieve new electronic communications.

[0539] Input: Mail server connection information and authentication information

[0540] Output: Newly received electronic communications

[0541] What it does: It uses the IMAP protocol to connect to a mail server, retrieve new electronic communications, and store them in a database.

[0542] Step 2:

[0543] The server uses a generation AI to analyze the acquired electronic communications and automatically generate a summary of the contents.

[0544] Input: Retrieved electronic communication

[0545] Output: Generated summary

[0546] Specific operation: Uses the OpenAI API to analyze the content of acquired electronic communications and generate a summary. For example, input the body of an email as a prompt and receive the summary result.

[0547] Step 3:

[0548] The server determines the importance of the generated summary.

[0549] Input: Generated summary and email metadata (sender, subject, etc.)

[0550] Output: Importance judgment result

[0551] Specific operation: An algorithm is run to determine the importance of an email based on the summary and the email's metadata, and past importance determination data is referenced.

[0552] Step 4:

[0553] The server refers to past response history, predicts the next action to be taken, and creates a response list.

[0554] Input: Generated summary and past correspondence history

[0555] Output: Correspondence list

[0556] Specific operation: Using generative AI, analyze the summary content and past response history, predict the appropriate next action to take, and generate a response list. Instructions are given to the AI ​​using prompt sentences.

[0557] Step 5:

[0558] The server transmits the correspondence list and the summary to the user terminal for display.

[0559] Input: Correspondence list and summary

[0560] Output: Corresponding list and summary displayed on user terminal

[0561] Specific operation: The server sends a correspondence list and summary to the user's smartphone or PC, and displays it using a dedicated application.

[0562] Step 6:

[0563] The user checks the summary and correspondence list displayed on the terminal and then performs the next action to be taken.

[0564] Input: Summary and corresponding list displayed on terminal

[0565] Output: Specific actions taken

[0566] Specific actions: The user checks the summary and correspondence list on the terminal and promptly takes necessary actions based on it (e.g., submitting troubleshooting instructions for the payment system).

[0567] The above processing steps realize a system that can quickly and efficiently identify important communications from among a vast amount of electronic communications and provide appropriate customer support.

[0568] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0569] This invention relates to a system that uses generative AI to analyze emails and automatically generate summaries of their contents, as well as a system that incorporates an emotion engine that recognizes the user's emotions, allowing the reevaluation of email importance and customization of action lists.

[0570] Overall system configuration

[0571] The system consists of the following components:

[0572] 1. Server: Receives emails, analyzes their contents, generates summaries, determines their importance, and creates action lists.

[0573] 2. User terminal: displays a summary and action list and monitors the user's emotional state.

[0574] 3. User: Uses the system to check and respond to emails.

[0575] Server Operation

[0576] The server connects to the specified mail server, retrieves, analyzes, summarizes, and determines the importance of emails based on their content.The server then refers to past response history and uses predictive AI to predict the next action to be taken, generating an action list.

[0577] Furthermore, the system uses an emotion engine to analyze the user's emotional state and reassess the importance of the email based on that information. Based on this reassessment, the system customizes the action list and prioritizes suggested responses according to the user's emotional state.

[0578] User terminal operation

[0579] The user device displays the summary and action list sent from the server in real time. The device is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice via a camera and microphone to recognize their emotional state.

[0580] This emotional state is monitored in real time and fed back to the server, which then dynamically adjusts the display format and content of the summary and action list.

[0581] How users use it

[0582] The user starts the system at the end of work and checks the summary of the day's received emails on the device. For example, an email from a boss asking for a project progress report is displayed in the summary as being of high importance. If the user's emotional state indicates "stress," the system will prioritize actions that help reduce stress, such as "relax before starting to write a progress report," in the action list.

[0583] Specifically, while a user is checking their email on the screen, the device's camera analyzes their facial expressions and the microphone analyzes their tone of voice to recognize their emotional state. Meanwhile, the server receives this data in real time and dynamically adjusts the email display and action list. By receiving this interactive feedback, users can respond to their work efficiently while taking their emotions into consideration.

[0584] Specifically, when a user is feeling stressed or anxious, the emotion engine can automatically prioritize email processing taking their emotions into consideration. It can also take actions or make suggestions to encourage relaxation as needed, reducing the user's mental burden.

[0585] In this way, by combining generative AI and an emotion engine, the present invention provides a novel system that improves the efficiency of email processing and subsequent actions while taking into account the user's emotional state.

[0586] The processing flow will be explained below.

[0587] Step 1:

[0588] The server connects to the mail server using the IMAP protocol and retrieves the emails received that day from the specified mailbox (usually the inbox), which are then stored in a database.

[0589] Step 2:

[0590] The server extracts the text body of each retrieved email, which includes determining whether the email is multi-part or single-part and extracting the text from the appropriate part.

[0591] Step 3:

[0592] The server then uses generative AI to analyze the extracted email text and automatically generate a summary of the email's contents, which concisely summarizes the main points of the email.

[0593] Step 4:

[0594] The server determines the importance of an email based on metadata such as the sender and subject line of the email. For example, an email from your boss or about an important project might be considered high importance.

[0595] Step 5:

[0596] The server combines the generated summary with the determined importance to generate summary data, which includes information about the sender, subject, summary, and importance of the email.

[0597] Step 6:

[0598] The device uses a camera and microphone to analyze the user's facial expressions and tone of voice in real time to recognize the user's emotional state, thereby determining the user's stress level and emotional state.

[0599] Step 7:

[0600] The server receives the user's emotional state data and re-evaluates the importance of the emails based on the data, and determines the order in which the emails are processed that is optimal for the user's current emotional state.

[0601] Step 8:

[0602] The server references past response history and uses predictive AI to predict the next action to be taken for the generated summary. Based on this prediction, a specific action list is created.

[0603] Step 9:

[0604] The server generates a customized action list taking into account the user's emotional state and sends it to the user's device along with summary data. Based on the user's emotional state, actions that help reduce stress and urgent emails are prioritized.

[0605] Step 10:

[0606] The terminal displays the summary data sent from the server and a customized action list, through which the user can check a summary of all emails, their importance, and the next action to be taken.

[0607] Step 11:

[0608] The user can take necessary actions based on the summary and action list displayed on the device, such as sending a progress report in response to an important email from a boss, or performing a relaxation action when stress levels are high.

[0609] Through this series of processes, the system takes into account the user's emotional state to improve the efficiency of email processing and subsequent actions, thereby reducing the user's mental burden.

[0610] Example 2

[0611] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0612] In today's work environment, there is a huge amount of electronic communication, and people need to be able to quickly extract important information from it and respond appropriately. However, this takes time and effort, and it is especially difficult to respond effectively when users are stressed or tired. Furthermore, many current systems do not take users' emotional state into account and can only provide fixed responses, which hinders users' efficient work performance.

[0613] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for analyzing electronic communications using a generation AI and automatically generating a summary of the contents, means for determining the importance of the electronic communications, means for displaying the summary based on the determined importance, means for predicting the next action to be taken by referring to past response history and creating a work instruction list, means for displaying the work instruction list and summary on a user terminal, means for monitoring the user's emotional state, reevaluating the importance based on that information, and customizing the work instruction list, and means for analyzing emotional data in real time and feeding it back to the server. This enables efficient information extraction and response that takes the user's emotional state into consideration.

[0614] "Generative AI" is a technology that uses artificial intelligence to generate and analyze data.

[0615] "Electronic communications" means digital information sent or received over the Internet or an intranet.

[0616] A summary is a concise summary of the important points of a longer piece of text or data.

[0617] "Importance" is a measure of the priority or importance associated with particular information or data.

[0618] "Past response history" refers to records of responses and processes that have been taken in the past.

[0619] A "work instruction list" is a list of actions and procedures required to carry out a specific job or task.

[0620] A "user terminal" is a device that a user directly operates to display and input information. Generally, this includes PCs and smartphones.

[0621] "Emotional state" is a term that refers to a user's psychological state or mood.

[0622] "Feedback" refers to information or data given regarding the behavior or results of a system.

[0623] This invention relates to a system that uses generative AI to analyze electronic communications and automatically generate summaries of their contents, and also incorporates an emotion engine that recognizes the user's emotions, enabling the reevaluation of the importance of communications and the customization of action instruction lists.

[0624] Server Operation

[0625] The server first connects to the designated mail server to receive the electronic communication, which is accomplished using standard protocols such as IMAP (Internet Message Access Protocol) or POP3 (Post Office Protocol 3).

[0626] Examples:

[0627] The server connects to a communication server using the IMAP protocol and obtains new communications.

[0628] The captured communications are stored in an internal database.

[0629] The server then uses a generative AI model (such as OpenAI's GPT-4) to summarize the content of the received communication, designing an appropriate prompt and passing it to the generative AI.

[0630] Example prompt sentence:

[0631] "Please summarize the contents of the following electronic communication: 'Details about the progress of the project...'"

[0632] Based on the generated summary, the server uses an algorithm to determine the importance of the communication, based on factors such as past response history and the frequency of occurrence of fixed keywords.

[0633] Furthermore, the server uses the prediction function of the generative AI to predict the next action to be taken and create a list of action instructions. For example, for a communication about "checking project progress," which is a high-priority item, it suggests a specific action such as "submit a progress report."

[0634] Device behavior

[0635] The user device displays the summary and action instruction list sent from the server in real time, and the device's built-in emotion engine analyzes the user's facial expressions and tone of voice via a camera and microphone to recognize their emotional state.

[0636] Examples:

[0637] While the user is checking the communication, the device's camera captures the user's facial expressions, and the emotion engine performs emotion analysis in real time.

[0638] The microphone analyzes the tone of the user's voice and extracts data on the level of stress and excitement.

[0639] The extracted emotion data is fed back to the server, which then uses this data to reevaluate the importance of the communication and the action instruction list, customizing them as necessary.

[0640] How users use it

[0641] The user starts the system at the end of work and checks a summary of the day's received communications on their device. For example, a communication from a superior about "checking project progress" is displayed in the summary as being of high importance. If the user's emotional state indicates "stress," the system will prioritize actions that help reduce stress, such as "relax before starting to write a progress report," in the list of action instructions.

[0642] Hardware and software used

[0643] The following hardware and software are used to implement this system:

[0644] Server: Server machine that handles communication (e.g. Linux server)

[0645] Generative AI models: such as OpenAI's GPT-4

[0646] Emotion engine: Software with the ability to analyze a user's facial expressions and voice (e.g., Microsoft Azure's Face API or Speech API)

[0647] User device: PC or smartphone equipped with a camera and microphone

[0648] Through the above steps, the present invention realizes a series of processes that analyzes electronic communications, generates summaries, and provides action instructions that take emotional states into account. By using this system, users can process information and respond to business tasks efficiently while taking their emotions into consideration.

[0649] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0650] Step 1:

[0651] The server receives new electronic communications from a designated mail server.

[0652] Specific behavior:

[0653] The server connects to the mail server using the IMAP or POP3 protocol.

[0654] Obtain a list of new electronic communications and store their details in an internal database.

[0655] Input: mail server connection information, query for new electronic communications.

[0656] Output: A list of retrieved new electronic communications.

[0657] Step 2:

[0658] The server uses the generative AI model to summarize the content of the received electronic communication.

[0659] Specific behavior:

[0660] The server passes the following prompt to the generative AI model:

[0661] "Please summarize the contents of the following electronic communication: 'Details about the progress of the project...'"

[0662] The generative AI model returns a summary and stores the results in an internal database.

[0663] Input: Content of received electronic communication, prompt text.

[0664] Output: The generated summary.

[0665] Step 3:

[0666] The server determines the importance of the email.

[0667] Specific behavior:

[0668] The server uses an importance determination algorithm to rate the importance of the received electronic communication.

[0669] The importance score is stored in an internal database.

[0670] Input: Received electronic communications, summaries, and past correspondence history.

[0671] Output: Importance score.

[0672] Step 4:

[0673] The server predicts what action to take next and creates an action list.

[0674] Specific behavior:

[0675] The server passes the following prompt to the generative AI model:

[0676] "Suggest next actions for high priority emails"

[0677] The generative AI model suggests specific actions and stores the results as an action list in an internal database.

[0678] Input: Importance score, prompt for the generation AI.

[0679] Output: Action list.

[0680] Step 5:

[0681] The device monitors the user's emotional state and provides feedback to the server.

[0682] Specific behavior:

[0683] The camera captures the user's facial expressions, and the emotion engine analyzes them.

[0684] The microphone analyzes the tone of the user's voice and extracts emotional data.

[0685] The emotion data is sent to the server.

[0686] Input: User's facial expression data, tone of voice data.

[0687] Output: Emotion data.

[0688] Step 6:

[0689] The server reevaluates and customizes the importance and action list of emails based on emotional data.

[0690] Specific behavior:

[0691] The server receives the emotion data and evaluates the importance and action list again.

[0692] If necessary, add customizations to the action list, such as "Relax and then continue working."

[0693] Save the updated action list to an internal database.

[0694] Input: sentiment data, original action list, importance score.

[0695] Output: The updated action list.

[0696] Step 7:

[0697] The terminal displays a summary and a customized action list to the user.

[0698] Specific behavior:

[0699] The device retrieves the summary and latest action list from the server.

[0700] Display a summary and action list on the user interface.

[0701] Input: Summary, action list from server.

[0702] Output: The summary and action list that is displayed to the user.

[0703] The above is the specific processing flow of this system. Each processing step works together to realize efficient analysis and response of electronic communications taking into account the user's emotional state.

[0704] (Application example 2)

[0705] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0706] In conventional electronic payment services, users had to manually check and respond to all notifications and transaction emails, which was cumbersome and often mentally taxing. Furthermore, because the system did not take into account the user's emotional state, there were issues such as missing important transactions in stressful situations or being unable to make appropriate decisions regarding high-risk transactions.

[0707] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing emails using a generation AI and automatically generating a summary of the contents, means for determining the importance of the email, means for displaying the summary based on the determined importance, means for predicting the next action to be taken by referring to past response history and creating an action list, means for displaying the action list and the summary on a user terminal, means for analyzing the user's emotional state using a camera and microphone, and means for reevaluating the importance based on the user's emotional state and customizing the action list. This enables the user to make optimal decisions taking into consideration their emotional state and efficiently process important notifications and transactions related to electronic payments.

[0708] "Generative AI" refers to technology that uses artificial intelligence models to automatically generate text and data.

[0709] "Email" is a means of communication for sending and receiving text information, files, etc. via the Internet.

[0710] The "summary" is information that briefly summarizes the contents of the email.

[0711] "Importance" is a metric used to assess the urgency and priority of an email or notification.

[0712] "Past response history" refers to a record of responses to emails that a user has made up to now.

[0713] An "action list" is a list of tasks that indicate the next action the user should take.

[0714] "User terminal" is a general term for electronic devices used by users, including smartphones and tablets.

[0715] A "camera" is a device that captures video or still images.

[0716] A "microphone" is a device that records sound.

[0717] An "emotional state" is an emotion a user is feeling at a particular point in time, and may include "stressed" or "relaxed."

[0718] "Analysis" is a method of examining data and information in detail and understanding its content.

[0719] "Reevaluation" means to repeat or review an evaluation that has already been made.

[0720] "Customization" refers to adjusting specific content or settings to suit the user's needs and circumstances.

[0721] This invention is a system that uses generative AI to analyze emails and automatically generate summaries of their contents, and also incorporates an emotion engine that recognizes the user's emotions, enabling the re-evaluation of email importance and customization of action lists. This system consists of the following components:

[0722] Server Operation

[0723] The server connects to the specified mail server, retrieves, analyzes, and summarizes emails, and determines their importance based on their content. It also references past response history to predict the next action to be taken and uses generative AI to create an action list. The server analyzes the user's emotional state through a camera or microphone, and based on that information, reevaluates the importance of emails and customizes the action list. Examples of generative AI models used include GPT-3 and BERT.

[0724] User terminal operation

[0725] The user's device displays the summary and action list sent from the server in real time. The device is equipped with an emotion engine that uses a camera and microphone to analyze the user's facial expressions and tone of voice to recognize their emotional state. This emotional state is fed back to the server in real time, enabling the device to respond in a way that takes the user's emotions into consideration.

[0726] Specific examples

[0727] For example, when a user launches the application at the end of work and checks the summary of the day's received emails on their device, an email from their boss asking them to "check the project progress" is displayed in the summary as being of high importance. If the user's emotional state indicates "stress," the system will prioritize actions that help reduce stress, such as "relax before starting to write a progress report," in the action list.

[0728] Hardware and software used

[0729] Camera: Used to capture the user's facial expressions.

[0730] Microphone: Used to analyze the tone of the user's voice.

[0731] Python standard library: used to retrieve and parse emails (smtplib, imaplib, email).

[0732] TextBlob: Used to summarize the email content.

[0733] Emotion Recognition: A library for analyzing user emotions.

[0734] Prompt Sentence Examples

[0735] Summarize emails about "payment notifications" and "important transaction confirmations," determine their importance based on the user's emotional state, and generate appropriate action suggestions. If the user's emotion is determined to be "stressed," suggest that they relax before proceeding.

[0736] This allows users to make optimal decisions while taking their emotions into consideration, and efficiently take appropriate actions regarding electronic payments and mail processing.

[0737] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0738] Step 1:

[0739] The server connects to the specified mail server and retrieves emails. It requires mail server login information as input and provides the retrieved email data as output.

[0740] Step 2:

[0741] The server analyzes the retrieved email and summarizes its contents. The email body is required as input, and a generative AI model (e.g., TextBlob) is used to generate a summary, resulting in a summary text as output.

[0742] Step 3:

[0743] The server determines the importance of an email. It takes parsed email metadata (sender, subject, etc.) and summary text as input, evaluates the importance using an importance determination algorithm, and outputs an importance score.

[0744] Step 4:

[0745] The server references past correspondence history and predicts the next action to be taken. Emails and past correspondence history data are required as input, and a generative AI model is used to make the prediction, resulting in a list of actions as output.

[0746] Step 5:

[0747] The server transmits the action list and summary to the user terminal. The generated action list and summary text are required as input, and data transmission to the user terminal is required as output.

[0748] Step 6:

[0749] The terminal displays the summary and action list sent from the server in real time. It requires data from the server as input and provides visual information to the user as output.

[0750] Step 7:

[0751] The device analyzes the user's emotional state using a camera and microphone. This process requires the user's facial expression and voice data as input, analyzes the emotional state using an emotion recognition algorithm, and obtains emotional data as output.

[0752] Step 8:

[0753] The server receives the user's emotional data, re-evaluates the importance of emails based on the data, and customizes the action list. The server requires the emotional data and the existing action list as input, adjusts the action list using a re-evaluation algorithm, and obtains the customized action list as output.

[0754] Step 9:

[0755] Again, the server sends the customized action list to the user terminal and suggests appropriate actions to the user. A customized action list is required as input, and the final action list is provided to the user as output.

[0756] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0757] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0758] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0759] [Third embodiment]

[0760] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0761] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0762] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0763] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0764] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0765] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0766] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0767] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0768] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0769] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0770] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0771] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0772] This invention relates to a system that uses generative AI to analyze emails, automatically generate a summary of their contents, and list the next actions to be taken.

[0773] Overall system structure

[0774] This system mainly consists of a server, a terminal, and a user. The server is responsible for receiving emails, generating summaries, determining importance, and creating action lists. The terminal provides an interface for users to check the summaries and action lists and perform necessary actions.

[0775] Server Operation

[0776] The server first connects to the specified mail server and retrieves all emails received that day. These emails are then stored in a database. Generative AI is then used to analyze these emails and summarize the content of each email. The summary is then ranked based on metadata such as the sender and subject.

[0777] Next, the server predicts the next action to be taken for the generated summary based on the past response history, and creates an action list based on this prediction result.

[0778] Device behavior

[0779] The terminal displays the summary and action list sent from the server. By checking this information on the terminal, the user can quickly grasp the summary and importance of all emails received that day. In addition, the user can check the next action to be taken as shown in the action list and take specific action based on it.

[0780] How users use it

[0781] At the end of work, a user starts the system and checks the summary of the day's received emails on their device. For example, a highly important email may be a request from a superior to "check project progress," and the summary may read, "Request for a report on the progress of Project X." In this case, the action list may include a specific action such as "Prepare and send a progress report." By following this instruction, the user can quickly take the necessary action, preventing email omissions and delays in response.

[0782] In a specific implementation, for example, the system connects to a mail server using the IMAP protocol and uses Python to retrieve and analyze email content. Furthermore, a natural language processing model is used as the generative AI, and the results are used to generate summaries and predict actions. This allows users to improve work efficiency and prevent oversight of important business emails.

[0783] The processing flow will be explained below.

[0784] Step 1:

[0785] The server connects to the specified mail server using the IMAP protocol, authenticates using the user's email account information (email address and password), and selects a mailbox (usually the inbox).

[0786] Step 2:

[0787] The server searches and retrieves all emails received that day. Based on the list of email IDs obtained as a search result, it retrieves detailed information about each email and stores it in a database.

[0788] Step 3:

[0789] The server extracts the text body of each retrieved email, which includes determining whether the email is multi-part or single-part and extracting the text from the appropriate part.

[0790] Step 4:

[0791] The server inputs the extracted email text into the generation AI, which automatically generates a summary of the content. The generated summary is temporarily saved along with metadata such as the email sender and subject.

[0792] Step 5:

[0793] The server determines the importance of each email based on metadata such as the sender and subject line of the email. For example, an email from your boss is considered high importance.

[0794] Step 6:

[0795] The server combines the generated summary with the determined importance to generate summary data of the email content, which includes information about the sender, subject, summary, and importance of the email.

[0796] Step 7:

[0797] The server references past response history and uses predictive AI to predict the next action to be taken for each summary. Based on the prediction results, a specific action list is created.

[0798] Step 8:

[0799] The server sends the generated summary data and action list to the user's device, transferring data in real time via a web server or API.

[0800] Step 9:

[0801] The terminal displays the summary data and action list sent from the server, and the user can check the summary and importance of all emails and the next action to be taken through the interface on the terminal.

[0802] Step 10:

[0803] The user can then take necessary action based on the displayed summary and action list through the terminal. For example, the user can create and reply to an important inquiry from a superior by creating a progress report.

[0804] Example 1

[0805] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0806] In today's information society, email is an important means of business communication. However, because a large volume of emails are received every day, sorting and responding to them requires a lot of time and effort. In particular, important emails can be overlooked or responded to late, which can reduce business efficiency. To solve this problem, a system is needed that can efficiently analyze emails, extract necessary information, and respond quickly.

[0807] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0808] In this invention, the server includes means for analyzing emails using a generative AI and automatically generating a summary of the contents, means for determining the importance of the email, means for displaying the summary based on the determined importance, means for predicting the next action to be taken by referring to past response history and creating an action list, means for displaying the action list and summary on a user terminal, means for connecting to a mail server using the IMAP protocol and retrieving received emails, and means for inputting prompt sentences into the generative AI model and analyzing emails using natural language processing. This allows users to efficiently manage received emails, prevent overlooking important emails or delaying responses, and improve work efficiency.

[0809] "Generative AI" is an artificial intelligence model that performs natural language processing based on input data and automatically generates sentences and information.

[0810] "Email" refers to a digital letter sent and received via the Internet or other means.

[0811] A "summary" is a short summary that extracts the main information from a long piece of text or data.

[0812] "Importance" is an indicator of how much priority a particular email should have.

[0813] An "action list" is a list of specific action items to be carried out according to a specific task or instruction.

[0814] "IMAP protocol" stands for Internet Message Access Protocol, and is a standard communication protocol used when receiving email from a mail server.

[0815] A "prompt" is an instruction or question entered into a generative AI model to instruct it to perform a specific task.

[0816] "Natural language processing" refers to the technology that allows computers to understand and process human language.

[0817] A "mail server" is a server that manages and stores the sending and receiving of e-mail.

[0818] A "database" is a system for efficiently searching, storing, and updating data.

[0819] A "user terminal" is a device (e.g., a computer or smartphone) that a user directly operates to access the system.

[0820] This invention relates to a system that uses generative AI to analyze emails, automatically summarize their contents, and create a list of next actions. The system of the present invention is mainly composed of a server, a terminal, and a user.

[0821] Server Operation

[0822] The server first connects to the mail server using the IMAP protocol and retrieves all emails received that day. The software for this is the Python language and its library, imaplib. The retrieved emails are then stored in a database. SQLite can be used as the database.

[0823] The server then uses a generative AI model (e.g., a natural language processing model) to analyze these emails and summarize the content of each email. The AI ​​model receives prompts and uses natural language processing to generate summaries.

[0824] Specific examples of prompts are as follows:

[0825] "Below is the content of a new email. Please summarize it, rate its importance, and suggest a next course of action based on that."

[0826] Email body:

[0827] "Progress on Project X is delayed, so we request an immediate response. Details will be explained at the meeting."

[0828] The generated summary is then ranked based on metadata such as the sender and subject line, and the server then uses past response history to predict the next course of action, creating an action list based on that.

[0829] Device behavior

[0830] The terminal displays the summary and action list sent from the server. A web browser is generally used as the user interface. JavaScript and HTML are used to enable the user to quickly grasp the summary and importance of all received emails.

[0831] How users use it

[0832] At the end of work, the user starts the system and checks the summary of the day's received emails on their device. For example, if a highly important email is a request for "project progress confirmation" and the summary reads "Requesting a report on the progress of Project X," the action list will include a specific action such as "Prepare and send a progress report." Users can follow these instructions to quickly take the necessary action, preventing emails from being overlooked or delaying responses.

[0833] In this way, the system of the present invention effectively analyzes emails using generative AI and provides users with useful summaries and specific action lists, thereby significantly improving the efficiency of their business management.

[0834] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0835] Step 1:

[0836] The server connects to the mail server using the IMAP protocol and retrieves all emails received that day. Specifically, it connects using the Python imaplib library and retrieves emails using the search method.

[0837] Input: Mail server account information (ID, password)

[0838] Output: Received email data

[0839] Specific behavior: The server establishes a connection to the mail server and retrieves all emails from the specified folder (e.g., inbox).

[0840] Step 2:

[0841] The server stores the retrieved emails in a database, which uses SQLite.

[0842] Input: Received email data

[0843] Output: Email data stored in a database (sender, subject, body, etc.)

[0844] Specific operation: The server analyzes the email data it receives, extracts information such as the sender, subject, and body of the email, and stores it in an SQLite database.

[0845] Step 3:

[0846] The server uses a generative AI model to analyze the stored emails and summarize the content of each one.

[0847] Input: Email body in database

[0848] Output: Generated summary

[0849] Specific operation: The server generates a prompt sentence for the AI ​​model, uses it as input for natural language processing, and generates a summary sentence.

[0850] Step 4:

[0851] The server determines the importance based on the content of the summarized summary.

[0852] Input: Generated summary

[0853] Output: Importance (high, low, etc.)

[0854] Specific operation: The server determines the importance by branching conditions based on keywords in the summary (e.g., "important," "urgent," etc.).

[0855] Step 5:

[0856] The server refers to the past response history, predicts the next action to be taken for the generated summary, and creates an action list.

[0857] Input: Generated summary, past correspondence history

[0858] Output: Action list

[0859] Specific operation: The server refers to a database of past cases and, based on similar past cases and response history, inputs prompts into the AI ​​model to predict the next action to be taken, generating the results as an action list.

[0860] Step 6:

[0861] The terminal displays a summary and an action list.

[0862] Input: Summary, Action List

[0863] Output: Summary and action list displayed in the user interface

[0864] Specific operation: The terminal displays the summary and action list through an interface such as a web browser, allowing the user to check them.

[0865] Step 7:

[0866] The user checks the summary and action list displayed on the terminal and takes the necessary action.

[0867] Input: Displayed summary and action list

[0868] Output: Actions taken (creating and sending progress reports, etc.)

[0869] Specific actions: The user operates the device to check the summary and action list and take action according to the instructions. For example, a specific action such as creating and sending a progress report is performed.

[0870] Through these specific processing steps, the system can efficiently manage received emails and prevent important emails from being overlooked or delayed in response.

[0871] (Application example 1)

[0872] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0873] With current electronic communications analysis systems, classifying and responding to a vast amount of communication usage requires time and effort, making it difficult to respond quickly and efficiently. Particularly in customer service operations, there is a risk of overlooking important communication usage, which can lead to delays in appropriate responses. It is also difficult to predict the next appropriate action to take based on response history, leading to lower customer satisfaction and a deterioration in work efficiency.

[0874] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0875] In this invention, the server includes means for analyzing electronic communications using a generation AI and automatically generating a summary of the content, means for determining the importance of the electronic communications, means for displaying the summary based on the determined importance, means for predicting the next action to be taken by referring to past response history and creating a response list, means for displaying the response list and summary on a user terminal, means for acquiring new electronic communications, means for automatically generating the next action to be taken from the summarized content and supporting customer responses, and means for summarizing and predicting actions using natural language processing with the generation AI. This makes it possible to quickly and efficiently identify important communication uses from a huge amount of electronic communications and provide appropriate customer responses.

[0876] "Generative AI" refers to technologies and systems that use artificial intelligence to generate data and information.

[0877] "Electronic communications" are digital forms of communication sent or received through email, messaging apps, etc.

[0878] A "summary" is a brief description of the contents of an electronic communication that abridges the content and extracts only the main points.

[0879] "Importance" is an indicator of how important the content of electronic communication is to business operations or responses.

[0880] "Target history" is a record of the responses and processes that have been carried out up to now.

[0881] An "action list" is a list of specific next steps to be taken based on the content of an electronic communication.

[0882] A "user terminal" is a device such as a computer or smart device used by a user.

[0883] "Natural language processing" is a technology that allows computers to understand natural language, and is a method for analyzing and generating text.

[0884] "Customer service" refers to the work that companies and organizations do to respond to inquiries and requests from customers.

[0885] This invention relates to a system that uses generative AI to analyze electronic communications, automatically generate a summary of the content, and list the next actions to be taken. The overall structure of this system mainly consists of a server, a terminal, and a user.

[0886] Server Operation

[0887] The server first connects to the specified mail server and retrieves new electronic communications. The retrieved electronic communications are stored in a database. Then, a generative AI is used to analyze these electronic communications and summarize the content of each communication. A natural language processing model is used as the generative AI. For example, OpenAI's API can be used.

[0888] The importance of the summarized communication content is determined based on metadata such as the sender and subject. Next, the server predicts the next action to be taken for the generated summary based on past response history, and creates a response list based on this prediction result.

[0889] Hardware and software used

[0890] Hardware: Servers, user devices (smartphones, PCs, etc.)

[0891] Software: Python, IMAP protocol, OpenAI API

[0892] Device behavior

[0893] The terminal displays the summary and action list sent from the server. The user can check this information on the terminal and quickly grasp the summary and importance of all electronic communications received that day. The user can also check the next action indicated in the action list and take specific action based on it. For example, this can be done using an application installed on a smartphone or head-mounted display.

[0894] How users use it

[0895] At the end of the workday, the user starts up the system and checks a summary of the day's incoming communications on their terminal. For example, if a high-priority communication is a customer inquiry about being unable to make a payment, and the summary reads, "We have received an inquiry about an issue preventing payment," the response list will include a specific action: "Send troubleshooting instructions for the payment system." Following these instructions, the user can quickly take the necessary action, preventing oversights and delays in customer support.

[0896] Prompt Sentence Examples

[0897] Summary generation prompt:

[0898] "Please summarize the email below:\n\nI'm inquiring about an inability to make a payment. What are the troubleshooting steps?"

[0899] Matching list generating prompt statement:

[0900] "Based on this summary, please list the next steps you should take:\n\nWe've received a call about an issue with being unable to make a payment."

[0901] This will realize a system that can quickly and efficiently identify important communications from among a vast amount of electronic communications and provide appropriate customer support.

[0902] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0903] Step 1:

[0904] The server connects to the designated mail server to retrieve new electronic communications.

[0905] Input: Mail server connection information and authentication information

[0906] Output: Newly received electronic communications

[0907] What it does: It uses the IMAP protocol to connect to a mail server, retrieve new electronic communications, and store them in a database.

[0908] Step 2:

[0909] The server uses a generation AI to analyze the acquired electronic communications and automatically generate a summary of the contents.

[0910] Input: Retrieved electronic communication

[0911] Output: Generated summary

[0912] Specific operation: Uses the OpenAI API to analyze the content of acquired electronic communications and generate a summary. For example, input the body of an email as a prompt and receive the summary result.

[0913] Step 3:

[0914] The server determines the importance of the generated summary.

[0915] Input: Generated summary and email metadata (sender, subject, etc.)

[0916] Output: Importance judgment result

[0917] Specific operation: An algorithm is run to determine the importance of an email based on the summary and the email's metadata, and past importance determination data is referenced.

[0918] Step 4:

[0919] The server refers to past response history, predicts the next action to be taken, and creates a response list.

[0920] Input: Generated summary and past correspondence history

[0921] Output: Correspondence list

[0922] Specific operation: Using generative AI, analyze the summary content and past response history, predict the appropriate next action to take, and generate a response list. Instructions are given to the AI ​​using prompt sentences.

[0923] Step 5:

[0924] The server transmits the correspondence list and the summary to the user terminal for display.

[0925] Input: Correspondence list and summary

[0926] Output: Corresponding list and summary displayed on user terminal

[0927] Specific operation: The server sends a correspondence list and summary to the user's smartphone or PC, and displays it using a dedicated application.

[0928] Step 6:

[0929] The user checks the summary and correspondence list displayed on the terminal and then performs the next action to be taken.

[0930] Input: Summary and corresponding list displayed on terminal

[0931] Output: Specific actions taken

[0932] Specific actions: The user checks the summary and correspondence list on the terminal and promptly takes necessary actions based on it (e.g., submitting troubleshooting instructions for the payment system).

[0933] The above processing steps realize a system that can quickly and efficiently identify important communications from among a vast amount of electronic communications and provide appropriate customer support.

[0934] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0935] This invention relates to a system that uses generative AI to analyze emails and automatically generate summaries of their contents, as well as a system that incorporates an emotion engine that recognizes the user's emotions, allowing the reevaluation of email importance and customization of action lists.

[0936] Overall system configuration

[0937] The system consists of the following components:

[0938] 1. Server: Receives emails, analyzes their contents, generates summaries, determines their importance, and creates action lists.

[0939] 2. User terminal: displays a summary and action list and monitors the user's emotional state.

[0940] 3. User: Uses the system to check and respond to emails.

[0941] Server Operation

[0942] The server connects to the specified mail server, retrieves, analyzes, summarizes, and determines the importance of emails based on their content.The server then refers to past response history and uses predictive AI to predict the next action to be taken, generating an action list.

[0943] Furthermore, the system uses an emotion engine to analyze the user's emotional state and reassess the importance of the email based on that information. Based on this reassessment, the system customizes the action list and prioritizes suggested responses according to the user's emotional state.

[0944] User terminal operation

[0945] The user device displays the summary and action list sent from the server in real time. The device is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice via a camera and microphone to recognize their emotional state.

[0946] This emotional state is monitored in real time and fed back to the server, which then dynamically adjusts the display format and content of the summary and action list.

[0947] How users use it

[0948] The user starts the system at the end of work and checks the summary of the day's received emails on the device. For example, an email from a boss asking for a project progress report is displayed in the summary as being of high importance. If the user's emotional state indicates "stress," the system will prioritize actions that help reduce stress, such as "relax before starting to write a progress report," in the action list.

[0949] Specifically, while a user is checking their email on the screen, the device's camera analyzes their facial expressions and the microphone analyzes their tone of voice to recognize their emotional state. Meanwhile, the server receives this data in real time and dynamically adjusts the email display and action list. By receiving this interactive feedback, users can respond to their work efficiently while taking their emotions into consideration.

[0950] Specifically, when a user is feeling stressed or anxious, the emotion engine can automatically prioritize email processing taking their emotions into consideration. It can also take actions or make suggestions to encourage relaxation as needed, reducing the user's mental burden.

[0951] In this way, by combining generative AI and an emotion engine, the present invention provides a novel system that improves the efficiency of email processing and subsequent actions while taking into account the user's emotional state.

[0952] The processing flow will be explained below.

[0953] Step 1:

[0954] The server connects to the mail server using the IMAP protocol and retrieves the emails received that day from the specified mailbox (usually the inbox), which are then stored in a database.

[0955] Step 2:

[0956] The server extracts the text body of each retrieved email, which includes determining whether the email is multi-part or single-part and extracting the text from the appropriate part.

[0957] Step 3:

[0958] The server then uses generative AI to analyze the extracted email text and automatically generate a summary of the email's contents, which concisely summarizes the main points of the email.

[0959] Step 4:

[0960] The server determines the importance of an email based on metadata such as the sender and subject line of the email. For example, an email from your boss or about an important project might be considered high importance.

[0961] Step 5:

[0962] The server combines the generated summary with the determined importance to generate summary data, which includes information about the sender, subject, summary, and importance of the email.

[0963] Step 6:

[0964] The device uses a camera and microphone to analyze the user's facial expressions and tone of voice in real time to recognize the user's emotional state, thereby determining the user's stress level and emotional state.

[0965] Step 7:

[0966] The server receives the user's emotional state data and re-evaluates the importance of the emails based on the data, and determines the order in which the emails are processed that is optimal for the user's current emotional state.

[0967] Step 8:

[0968] The server references past response history and uses predictive AI to predict the next action to be taken for the generated summary. Based on this prediction, a specific action list is created.

[0969] Step 9:

[0970] The server generates a customized action list taking into account the user's emotional state and sends it to the user's device along with summary data. Based on the user's emotional state, actions that help reduce stress and urgent emails are prioritized.

[0971] Step 10:

[0972] The terminal displays the summary data sent from the server and a customized action list, through which the user can check a summary of all emails, their importance, and the next action to be taken.

[0973] Step 11:

[0974] The user can take necessary actions based on the summary and action list displayed on the device, such as sending a progress report in response to an important email from a boss, or performing a relaxation action when stress levels are high.

[0975] Through this series of processes, the system takes into account the user's emotional state to improve the efficiency of email processing and subsequent actions, thereby reducing the user's mental burden.

[0976] Example 2

[0977] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0978] In today's work environment, there is a huge amount of electronic communication, and people need to be able to quickly extract important information from it and respond appropriately. However, this takes time and effort, and it is especially difficult to respond effectively when users are stressed or tired. Furthermore, many current systems do not take users' emotional state into account and can only provide fixed responses, which hinders users' efficient work performance.

[0979] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for analyzing electronic communications using a generation AI and automatically generating a summary of the contents, means for determining the importance of the electronic communications, means for displaying the summary based on the determined importance, means for predicting the next action to be taken by referring to past response history and creating a work instruction list, means for displaying the work instruction list and summary on a user terminal, means for monitoring the user's emotional state, reevaluating the importance based on that information, and customizing the work instruction list, and means for analyzing emotional data in real time and feeding it back to the server. This enables efficient information extraction and response that takes the user's emotional state into consideration.

[0980] "Generative AI" is a technology that uses artificial intelligence to generate and analyze data.

[0981] "Electronic communications" means digital information sent or received over the Internet or an intranet.

[0982] A summary is a concise summary of the important points of a longer piece of text or data.

[0983] "Importance" is a measure of the priority or importance associated with particular information or data.

[0984] "Past response history" refers to records of responses and processes that have been taken in the past.

[0985] A "work instruction list" is a list of actions and procedures required to carry out a specific job or task.

[0986] A "user terminal" is a device that a user directly operates to display and input information. Generally, this includes PCs and smartphones.

[0987] "Emotional state" is a term that refers to a user's psychological state or mood.

[0988] "Feedback" refers to information or data given regarding the behavior or results of a system.

[0989] This invention relates to a system that uses generative AI to analyze electronic communications and automatically generate summaries of their contents, and also incorporates an emotion engine that recognizes the user's emotions, enabling the reevaluation of the importance of communications and the customization of action instruction lists.

[0990] Server Operation

[0991] The server first connects to the designated mail server to receive the electronic communication, which is accomplished using standard protocols such as IMAP (Internet Message Access Protocol) or POP3 (Post Office Protocol 3).

[0992] Examples:

[0993] The server connects to a communication server using the IMAP protocol and obtains new communications.

[0994] The captured communications are stored in an internal database.

[0995] The server then uses a generative AI model (such as OpenAI's GPT-4) to summarize the content of the received communication, designing an appropriate prompt and passing it to the generative AI.

[0996] Example prompt sentence:

[0997] "Please summarize the contents of the following electronic communication: 'Details about the progress of the project...'"

[0998] Based on the generated summary, the server uses an algorithm to determine the importance of the communication, based on factors such as past response history and the frequency of occurrence of fixed keywords.

[0999] Furthermore, the server uses the prediction function of the generative AI to predict the next action to be taken and create a list of action instructions. For example, for a communication about "checking project progress," which is a high-priority item, it suggests a specific action such as "submit a progress report."

[1000] Device behavior

[1001] The user device displays the summary and action instruction list sent from the server in real time, and the device's built-in emotion engine analyzes the user's facial expressions and tone of voice via a camera and microphone to recognize their emotional state.

[1002] Examples:

[1003] While the user is checking the communication, the device's camera captures the user's facial expressions, and the emotion engine performs emotion analysis in real time.

[1004] The microphone analyzes the tone of the user's voice and extracts data on the level of stress and excitement.

[1005] The extracted emotion data is fed back to the server, which then uses this data to reevaluate the importance of the communication and the action instruction list, customizing them as necessary.

[1006] How users use it

[1007] The user starts the system at the end of work and checks a summary of the day's received communications on their device. For example, a communication from a superior about "checking project progress" is displayed in the summary as being of high importance. If the user's emotional state indicates "stress," the system will prioritize actions that help reduce stress, such as "relax before starting to write a progress report," in the list of action instructions.

[1008] Hardware and software used

[1009] The following hardware and software are used to implement this system:

[1010] Server: Server machine that handles communication (e.g. Linux server)

[1011] Generative AI models: such as OpenAI's GPT-4

[1012] Emotion engine: Software with the ability to analyze a user's facial expressions and voice (e.g., Microsoft Azure's Face API or Speech API)

[1013] User device: PC or smartphone equipped with a camera and microphone

[1014] Through the above steps, the present invention realizes a series of processes that analyzes electronic communications, generates summaries, and provides action instructions that take emotional states into account. By using this system, users can process information and respond to business tasks efficiently while taking their emotions into consideration.

[1015] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1016] Step 1:

[1017] The server receives new electronic communications from a designated mail server.

[1018] Specific behavior:

[1019] The server connects to the mail server using the IMAP or POP3 protocol.

[1020] Obtain a list of new electronic communications and store their details in an internal database.

[1021] Input: mail server connection information, query for new electronic communications.

[1022] Output: A list of retrieved new electronic communications.

[1023] Step 2:

[1024] The server uses the generative AI model to summarize the content of the received electronic communication.

[1025] Specific behavior:

[1026] The server passes the following prompt to the generative AI model:

[1027] "Please summarize the contents of the following electronic communication: 'Details about the progress of the project...'"

[1028] The generative AI model returns a summary and stores the results in an internal database.

[1029] Input: Content of received electronic communication, prompt text.

[1030] Output: The generated summary.

[1031] Step 3:

[1032] The server determines the importance of the email.

[1033] Specific behavior:

[1034] The server uses an importance determination algorithm to rate the importance of the received electronic communication.

[1035] The importance score is stored in an internal database.

[1036] Input: Received electronic communications, summaries, and past correspondence history.

[1037] Output: Importance score.

[1038] Step 4:

[1039] The server predicts what action to take next and creates an action list.

[1040] Specific behavior:

[1041] The server passes the following prompt to the generative AI model:

[1042] "Suggest next actions for high priority emails"

[1043] The generative AI model suggests specific actions and stores the results as an action list in an internal database.

[1044] Input: Importance score, prompt for the generation AI.

[1045] Output: Action list.

[1046] Step 5:

[1047] The device monitors the user's emotional state and provides feedback to the server.

[1048] Specific behavior:

[1049] The camera captures the user's facial expressions, and the emotion engine analyzes them.

[1050] The microphone analyzes the tone of the user's voice and extracts emotional data.

[1051] The emotion data is sent to the server.

[1052] Input: User's facial expression data, tone of voice data.

[1053] Output: Emotion data.

[1054] Step 6:

[1055] The server reevaluates and customizes the importance and action list of emails based on emotional data.

[1056] Specific behavior:

[1057] The server receives the emotion data and evaluates the importance and action list again.

[1058] If necessary, add customizations to the action list, such as "Relax and then continue working."

[1059] Save the updated action list to an internal database.

[1060] Input: sentiment data, original action list, importance score.

[1061] Output: The updated action list.

[1062] Step 7:

[1063] The terminal displays a summary and a customized action list to the user.

[1064] Specific behavior:

[1065] The device retrieves the summary and latest action list from the server.

[1066] Display a summary and action list on the user interface.

[1067] Input: Summary, action list from server.

[1068] Output: The summary and action list that is displayed to the user.

[1069] The above is the specific processing flow of this system. Each processing step works together to realize efficient analysis and response of electronic communications taking into account the user's emotional state.

[1070] (Application example 2)

[1071] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1072] In conventional electronic payment services, users had to manually check and respond to all notifications and transaction emails, which was cumbersome and often mentally taxing. Furthermore, because the system did not take into account the user's emotional state, there were issues such as missing important transactions in stressful situations or being unable to make appropriate decisions regarding high-risk transactions.

[1073] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing emails using a generation AI and automatically generating a summary of the contents, means for determining the importance of the email, means for displaying the summary based on the determined importance, means for predicting the next action to be taken by referring to past response history and creating an action list, means for displaying the action list and the summary on a user terminal, means for analyzing the user's emotional state using a camera and microphone, and means for reevaluating the importance based on the user's emotional state and customizing the action list. This enables the user to make optimal decisions taking into consideration their emotional state and efficiently process important notifications and transactions related to electronic payments.

[1074] "Generative AI" refers to technology that uses artificial intelligence models to automatically generate text and data.

[1075] "Email" is a means of communication for sending and receiving text information, files, etc. via the Internet.

[1076] The "summary" is information that briefly summarizes the contents of the email.

[1077] "Importance" is a metric used to assess the urgency and priority of an email or notification.

[1078] "Past response history" refers to a record of responses to emails that a user has made up to now.

[1079] An "action list" is a list of tasks that indicate the next action the user should take.

[1080] "User terminal" is a general term for electronic devices used by users, including smartphones and tablets.

[1081] A "camera" is a device that captures video or still images.

[1082] A "microphone" is a device that records sound.

[1083] An "emotional state" is an emotion a user is feeling at a particular point in time, and may include "stressed" or "relaxed."

[1084] "Analysis" is a method of examining data and information in detail and understanding its content.

[1085] "Reevaluation" means to repeat or review an evaluation that has already been made.

[1086] "Customization" refers to adjusting specific content or settings to suit the user's needs and circumstances.

[1087] This invention is a system that uses generative AI to analyze emails and automatically generate summaries of their contents, and also incorporates an emotion engine that recognizes the user's emotions, enabling the re-evaluation of email importance and customization of action lists. This system consists of the following components:

[1088] Server Operation

[1089] The server connects to the specified mail server, retrieves, analyzes, and summarizes emails, and determines their importance based on their content. It also references past response history to predict the next action to be taken and uses generative AI to create an action list. The server analyzes the user's emotional state through a camera or microphone, and based on that information, reevaluates the importance of emails and customizes the action list. Examples of generative AI models used include GPT-3 and BERT.

[1090] User terminal operation

[1091] The user's device displays the summary and action list sent from the server in real time. The device is equipped with an emotion engine that uses a camera and microphone to analyze the user's facial expressions and tone of voice to recognize their emotional state. This emotional state is fed back to the server in real time, enabling the device to respond in a way that takes the user's emotions into consideration.

[1092] Specific examples

[1093] For example, when a user launches the application at the end of work and checks the summary of the day's received emails on their device, an email from their boss asking them to "check the project progress" is displayed in the summary as being of high importance. If the user's emotional state indicates "stress," the system will prioritize actions that help reduce stress, such as "relax before starting to write a progress report," in the action list.

[1094] Hardware and software used

[1095] Camera: Used to capture the user's facial expressions.

[1096] Microphone: Used to analyze the tone of the user's voice.

[1097] Python standard library: used to retrieve and parse emails (smtplib, imaplib, email).

[1098] TextBlob: Used to summarize the email content.

[1099] Emotion Recognition: A library for analyzing user emotions.

[1100] Prompt Sentence Examples

[1101] Summarize emails about "payment notifications" and "important transaction confirmations," determine their importance based on the user's emotional state, and generate appropriate action suggestions. If the user's emotion is determined to be "stressed," suggest that they relax before proceeding.

[1102] This allows users to make optimal decisions while taking their emotions into consideration, and efficiently take appropriate actions regarding electronic payments and mail processing.

[1103] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1104] Step 1:

[1105] The server connects to the specified mail server and retrieves emails. It requires mail server login information as input and provides the retrieved email data as output.

[1106] Step 2:

[1107] The server analyzes the retrieved email and summarizes its contents. The email body is required as input, and a generative AI model (e.g., TextBlob) is used to generate a summary, resulting in a summary text as output.

[1108] Step 3:

[1109] The server determines the importance of an email. It takes parsed email metadata (sender, subject, etc.) and summary text as input, evaluates the importance using an importance determination algorithm, and outputs an importance score.

[1110] Step 4:

[1111] The server references past correspondence history and predicts the next action to be taken. Emails and past correspondence history data are required as input, and a generative AI model is used to make the prediction, resulting in a list of actions as output.

[1112] Step 5:

[1113] The server transmits the action list and summary to the user terminal. The generated action list and summary text are required as input, and data transmission to the user terminal is required as output.

[1114] Step 6:

[1115] The terminal displays the summary and action list sent from the server in real time. It requires data from the server as input and provides visual information to the user as output.

[1116] Step 7:

[1117] The device analyzes the user's emotional state using a camera and microphone. This process requires the user's facial expression and voice data as input, analyzes the emotional state using an emotion recognition algorithm, and obtains emotional data as output.

[1118] Step 8:

[1119] The server receives the user's emotional data, re-evaluates the importance of emails based on the data, and customizes the action list. The server requires the emotional data and the existing action list as input, adjusts the action list using a re-evaluation algorithm, and obtains the customized action list as output.

[1120] Step 9:

[1121] Again, the server sends the customized action list to the user terminal and suggests appropriate actions to the user. A customized action list is required as input, and the final action list is provided to the user as output.

[1122] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1123] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1124] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1125] [Fourth embodiment]

[1126] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1127] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1128] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1129] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1130] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1131] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1132] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1133] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1134] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1135] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1136] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1137] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1138] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1139] This invention relates to a system that uses generative AI to analyze emails, automatically generate a summary of their contents, and list the next actions to be taken.

[1140] Overall system structure

[1141] This system mainly consists of a server, a terminal, and a user. The server is responsible for receiving emails, generating summaries, determining importance, and creating action lists. The terminal provides an interface for users to check the summaries and action lists and perform necessary actions.

[1142] Server Operation

[1143] The server first connects to the specified mail server and retrieves all emails received that day. These emails are then stored in a database. Generative AI is then used to analyze these emails and summarize the content of each email. The summary is then ranked based on metadata such as the sender and subject.

[1144] Next, the server predicts the next action to be taken for the generated summary based on the past response history, and creates an action list based on this prediction result.

[1145] Device behavior

[1146] The terminal displays the summary and action list sent from the server. By checking this information on the terminal, the user can quickly grasp the summary and importance of all emails received that day. In addition, the user can check the next action to be taken as shown in the action list and take specific action based on it.

[1147] How users use it

[1148] At the end of work, a user starts the system and checks the summary of the day's received emails on their device. For example, a highly important email may be a request from a superior to "check project progress," and the summary may read, "Request for a report on the progress of Project X." In this case, the action list may include a specific action such as "Prepare and send a progress report." By following this instruction, the user can quickly take the necessary action, preventing email omissions and delays in response.

[1149] In a specific implementation, for example, the system connects to a mail server using the IMAP protocol and uses Python to retrieve and analyze email content. Furthermore, a natural language processing model is used as the generative AI, and the results are used to generate summaries and predict actions. This allows users to improve work efficiency and prevent oversight of important business emails.

[1150] The processing flow will be explained below.

[1151] Step 1:

[1152] The server connects to the specified mail server using the IMAP protocol, authenticates using the user's email account information (email address and password), and selects a mailbox (usually the inbox).

[1153] Step 2:

[1154] The server searches and retrieves all emails received that day. Based on the list of email IDs obtained as a search result, it retrieves detailed information about each email and stores it in a database.

[1155] Step 3:

[1156] The server extracts the text body of each retrieved email, which includes determining whether the email is multi-part or single-part and extracting the text from the appropriate part.

[1157] Step 4:

[1158] The server inputs the extracted email text into the generation AI, which automatically generates a summary of the content. The generated summary is temporarily saved along with metadata such as the email sender and subject.

[1159] Step 5:

[1160] The server determines the importance of each email based on metadata such as the sender and subject line of the email. For example, an email from your boss is considered high importance.

[1161] Step 6:

[1162] The server combines the generated summary with the determined importance to generate summary data of the email content, which includes information about the email sender, subject, summary, and importance.

[1163] Step 7:

[1164] The server references past response history and uses predictive AI to predict the next action to be taken for each summary. Based on the prediction results, a specific action list is created.

[1165] Step 8:

[1166] The server sends the generated summary data and action list to the user's device, transferring data in real time via a web server or API.

[1167] Step 9:

[1168] The terminal displays the summary data and action list sent from the server, and the user can check the summary and importance of all emails and the next action to be taken through the interface on the terminal.

[1169] Step 10:

[1170] The user can then take necessary action based on the displayed summary and action list through the terminal. For example, the user can create and reply to an important inquiry from a superior by creating a progress report.

[1171] Example 1

[1172] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1173] In today's information society, email is an important means of business communication. However, because a large volume of emails are received every day, sorting and responding to them requires a lot of time and effort. In particular, important emails can be overlooked or responded to late, which can reduce business efficiency. To solve this problem, a system is needed that can efficiently analyze emails, extract necessary information, and respond quickly.

[1174] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1175] In this invention, the server includes means for analyzing emails using a generative AI and automatically generating a summary of the contents, means for determining the importance of the email, means for displaying the summary based on the determined importance, means for predicting the next action to be taken by referring to past response history and creating an action list, means for displaying the action list and summary on a user terminal, means for connecting to a mail server using the IMAP protocol and retrieving received emails, and means for inputting prompt sentences into the generative AI model and analyzing emails using natural language processing. This allows users to efficiently manage received emails, prevent overlooking important emails or delaying responses, and improve work efficiency.

[1176] "Generative AI" is an artificial intelligence model that performs natural language processing based on input data and automatically generates sentences and information.

[1177] "Email" refers to a digital letter sent and received via the Internet or other means.

[1178] A "summary" is a short summary that extracts the main information from a long piece of text or data.

[1179] "Importance" is an indicator of how much priority a particular email should have.

[1180] An "action list" is a list of specific action items to be carried out according to a specific task or instruction.

[1181] "IMAP protocol" stands for Internet Message Access Protocol, and is a standard communication protocol used when receiving email from a mail server.

[1182] A "prompt" is an instruction or question entered into a generative AI model to instruct it to perform a specific task.

[1183] "Natural language processing" refers to the technology that allows computers to understand and process human language.

[1184] A "mail server" is a server that manages and stores the sending and receiving of e-mail.

[1185] A "database" is a system for efficiently searching, storing, and updating data.

[1186] A "user terminal" is a device (e.g., a computer or smartphone) that a user directly operates to access the system.

[1187] This invention relates to a system that uses generative AI to analyze emails, automatically summarize their contents, and create a list of next actions. The system of the present invention is mainly composed of a server, a terminal, and a user.

[1188] Server Operation

[1189] The server first connects to the mail server using the IMAP protocol and retrieves all emails received that day. The software for this is the Python language and its library, imaplib. The retrieved emails are then stored in a database. SQLite can be used as the database.

[1190] The server then uses a generative AI model (e.g., a natural language processing model) to analyze these emails and summarize the content of each email. The AI ​​model receives prompts and uses natural language processing to generate summaries.

[1191] Specific examples of prompts are as follows:

[1192] "Below is the content of a new email. Please summarize it, rate its importance, and suggest a next course of action based on that."

[1193] Email body:

[1194] "Progress on Project X is delayed, so we request an immediate response. Details will be explained at the meeting."

[1195] The generated summary is then ranked based on metadata such as the sender and subject line, and the server then uses past response history to predict the next course of action, creating an action list based on that.

[1196] Device behavior

[1197] The terminal displays the summary and action list sent from the server. A web browser is generally used as the user interface. JavaScript and HTML are used to enable the user to quickly grasp the summary and importance of all received emails.

[1198] How users use it

[1199] At the end of work, the user starts the system and checks the summary of the day's received emails on their device. For example, if a highly important email is a request for "project progress confirmation" and the summary reads "Requesting a report on the progress of Project X," the action list will include a specific action such as "Prepare and send a progress report." Users can follow these instructions to quickly take the necessary action, preventing emails from being overlooked or delaying responses.

[1200] In this way, the system of the present invention effectively analyzes emails using generative AI and provides users with useful summaries and specific action lists, thereby significantly improving the efficiency of their business management.

[1201] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1202] Step 1:

[1203] The server connects to the mail server using the IMAP protocol and retrieves all emails received that day. Specifically, it connects using the Python imaplib library and retrieves emails using the search method.

[1204] Input: Mail server account information (ID, password)

[1205] Output: Received email data

[1206] Specific behavior: The server establishes a connection to the mail server and retrieves all emails from the specified folder (e.g., inbox).

[1207] Step 2:

[1208] The server stores the retrieved emails in a database, which uses SQLite.

[1209] Input: Received email data

[1210] Output: Email data stored in a database (sender, subject, body, etc.)

[1211] Specific operation: The server analyzes the email data it receives, extracts information such as the sender, subject, and body of the email, and stores it in an SQLite database.

[1212] Step 3:

[1213] The server uses a generative AI model to analyze the stored emails and summarize the content of each one.

[1214] Input: Email body in database

[1215] Output: Generated summary

[1216] Specific operation: The server generates a prompt sentence for the AI ​​model, uses it as input for natural language processing, and generates a summary sentence.

[1217] Step 4:

[1218] The server determines the importance based on the content of the summarized summary.

[1219] Input: Generated summary

[1220] Output: Importance (high, low, etc.)

[1221] Specific operation: The server determines the importance by branching conditions based on keywords in the summary (e.g., "important," "urgent," etc.).

[1222] Step 5:

[1223] The server refers to the past response history, predicts the next action to be taken for the generated summary, and creates an action list.

[1224] Input: Generated summary, past correspondence history

[1225] Output: Action list

[1226] Specific operation: The server refers to a database of past cases and, based on similar past cases and response history, inputs prompts into the AI ​​model to predict the next action to be taken, generating the results as an action list.

[1227] Step 6:

[1228] The terminal displays a summary and an action list.

[1229] Input: Summary, Action List

[1230] Output: Summary and action list displayed in the user interface

[1231] Specific operation: The terminal displays the summary and action list through an interface such as a web browser, allowing the user to check them.

[1232] Step 7:

[1233] The user checks the summary and action list displayed on the terminal and takes the necessary action.

[1234] Input: Displayed summary and action list

[1235] Output: Actions taken (creating and sending progress reports, etc.)

[1236] Specific actions: The user operates the device to check the summary and action list and take action according to the instructions. For example, a specific action such as creating and sending a progress report is performed.

[1237] Through these specific processing steps, the system can efficiently manage received emails and prevent important emails from being overlooked or delayed in response.

[1238] (Application example 1)

[1239] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1240] With current electronic communications analysis systems, classifying and responding to a vast amount of communication usage requires time and effort, making it difficult to respond quickly and efficiently. Particularly in customer service operations, there is a risk of overlooking important communication usage, which can lead to delays in appropriate responses. It is also difficult to predict the next appropriate action to take based on response history, leading to lower customer satisfaction and a deterioration in work efficiency.

[1241] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1242] In this invention, the server includes means for analyzing electronic communications using a generation AI and automatically generating a summary of the content, means for determining the importance of the electronic communications, means for displaying the summary based on the determined importance, means for predicting the next action to be taken by referring to past response history and creating a response list, means for displaying the response list and summary on a user terminal, means for acquiring new electronic communications, means for automatically generating the next action to be taken from the summarized content and supporting customer responses, and means for summarizing and predicting actions using natural language processing with the generation AI. This makes it possible to quickly and efficiently identify important communication uses from a huge amount of electronic communications and provide appropriate customer responses.

[1243] "Generative AI" refers to technologies and systems that use artificial intelligence to generate data and information.

[1244] "Electronic communications" are digital forms of communication sent or received through email, messaging apps, etc.

[1245] A "summary" is a brief description of the contents of an electronic communication that abridges the content and extracts only the main points.

[1246] "Importance" is an indicator of how important the content of electronic communication is to business operations or responses.

[1247] "Target history" is a record of the responses and processes that have been carried out up to now.

[1248] An "action list" is a list of specific next steps to be taken based on the content of an electronic communication.

[1249] A "user terminal" is a device such as a computer or smart device used by a user.

[1250] "Natural language processing" is a technology that allows computers to understand natural language, and is a method for analyzing and generating text.

[1251] "Customer service" refers to the work that companies and organizations do to respond to inquiries and requests from customers.

[1252] This invention relates to a system that uses generative AI to analyze electronic communications, automatically generate a summary of the content, and list the next actions to be taken. The overall structure of this system mainly consists of a server, a terminal, and a user.

[1253] Server Operation

[1254] The server first connects to the specified mail server and retrieves new electronic communications. The retrieved electronic communications are stored in a database. Then, a generative AI is used to analyze these electronic communications and summarize the content of each communication. A natural language processing model is used as the generative AI. For example, OpenAI's API can be used.

[1255] The importance of the summarized communication content is determined based on metadata such as the sender and subject.The server then predicts the next action to be taken for the generated summary based on past response history, and creates a response list based on this prediction.

[1256] Hardware and software used

[1257] Hardware: Servers, user devices (smartphones, PCs, etc.)

[1258] Software: Python, IMAP protocol, OpenAI API

[1259] Device behavior

[1260] The terminal displays the summary and action list sent from the server. The user can check this information on the terminal and quickly grasp the summary and importance of all electronic communications received that day. The user can also check the next action indicated in the action list and take specific action based on it. For example, this can be done using an application installed on a smartphone or head-mounted display.

[1261] How users use it

[1262] At the end of the workday, the user starts up the system and checks a summary of the day's incoming communications on their terminal. For example, if a high-priority communication is a customer inquiry about being unable to make a payment, and the summary reads, "We have received an inquiry about an issue preventing payment," the response list will include a specific action: "Send troubleshooting instructions for the payment system." Following these instructions, the user can quickly take the necessary action, preventing oversights and delays in customer support.

[1263] Prompt Sentence Examples

[1264] Summary generation prompt:

[1265] "Please summarize the email below:\n\nI'm inquiring about an inability to make a payment. What are the troubleshooting steps?"

[1266] Matching list generating prompt statement:

[1267] "Based on this summary, please list the next steps you should take:\n\nWe've received a call about an issue with being unable to make a payment."

[1268] This will realize a system that can quickly and efficiently identify important communications from among a vast amount of electronic communications and provide appropriate customer support.

[1269] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1270] Step 1:

[1271] The server connects to the designated mail server to retrieve new electronic communications.

[1272] Input: Mail server connection information and authentication information

[1273] Output: Newly received electronic communications

[1274] What it does: It uses the IMAP protocol to connect to a mail server, retrieve new electronic communications, and store them in a database.

[1275] Step 2:

[1276] The server uses a generation AI to analyze the acquired electronic communications and automatically generate a summary of the contents.

[1277] Input: Retrieved electronic communication

[1278] Output: Generated summary

[1279] Specific operation: Uses the OpenAI API to analyze the content of acquired electronic communications and generate a summary. For example, input the body of an email as a prompt and receive the summary result.

[1280] Step 3:

[1281] The server determines the importance of the generated summary.

[1282] Input: Generated summary and email metadata (sender, subject, etc.)

[1283] Output: Importance judgment result

[1284] Specific operation: An algorithm is run to determine the importance of an email based on the summary and the email's metadata, and past importance determination data is referenced.

[1285] Step 4:

[1286] The server refers to past response history, predicts the next action to be taken, and creates a response list.

[1287] Input: Generated summary and past correspondence history

[1288] Output: Correspondence list

[1289] Specific operation: Using generative AI, analyze the summary content and past response history, predict the appropriate next action to take, and generate a response list. Instructions are given to the AI ​​using prompt sentences.

[1290] Step 5:

[1291] The server transmits the correspondence list and the summary to the user terminal for display.

[1292] Input: Correspondence list and summary

[1293] Output: Corresponding list and summary displayed on user terminal

[1294] Specific operation: The server sends a correspondence list and summary to the user's smartphone or PC, and displays it using a dedicated application.

[1295] Step 6:

[1296] The user checks the summary and correspondence list displayed on the terminal and then performs the next action to be taken.

[1297] Input: Summary and corresponding list displayed on terminal

[1298] Output: Specific actions taken

[1299] Specific actions: The user checks the summary and correspondence list on the terminal and promptly takes necessary actions based on it (e.g., submitting troubleshooting instructions for the payment system).

[1300] The above processing steps realize a system that can quickly and efficiently identify important communications from among a vast amount of electronic communications and provide appropriate customer support.

[1301] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1302] This invention relates to a system that uses generative AI to analyze emails and automatically generate summaries of their contents, as well as a system that incorporates an emotion engine that recognizes the user's emotions, allowing the user to reassess the importance of emails and customize action lists.

[1303] Overall system configuration

[1304] The system consists of the following components:

[1305] 1. Server: Receives emails, analyzes their contents, generates summaries, determines their importance, and creates action lists.

[1306] 2. User terminal: displays a summary and action list and monitors the user's emotional state.

[1307] 3. User: Uses the system to check and respond to emails.

[1308] Server Operation

[1309] The server connects to the specified mail server, retrieves, analyzes, summarizes, and determines the importance of emails based on their content.The server then refers to past response history and uses predictive AI to predict the next action to be taken, generating an action list.

[1310] Furthermore, the system uses an emotion engine to analyze the user's emotional state and reassess the importance of the email based on that information. Based on this reassessment, the system customizes the action list and prioritizes responses that correspond to the user's emotional state.

[1311] User terminal operation

[1312] The user device displays the summary and action list sent from the server in real time. The device is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice via a camera and microphone to recognize their emotional state.

[1313] This emotional state is monitored in real time and fed back to the server, which then dynamically adjusts the display format and content of the summary and action list.

[1314] How users use it

[1315] The user starts the system at the end of work and checks the summary of the day's received emails on the device. For example, an email from a boss asking for a project progress report is displayed in the summary as being of high importance. If the user's emotional state indicates "stress," the system will prioritize actions that help reduce stress, such as "relax before starting to write a progress report," in the action list.

[1316] Specifically, while a user is checking their email on the screen, the device's camera analyzes their facial expressions and the microphone analyzes their tone of voice to recognize their emotional state. Meanwhile, the server receives this data in real time and dynamically adjusts the email display and action list. By receiving this interactive feedback, users can respond to their work efficiently while taking their emotions into consideration.

[1317] Specifically, when a user is feeling stressed or anxious, the emotion engine can automatically prioritize email processing taking their emotions into consideration. It can also take actions or make suggestions to encourage relaxation as needed, reducing the user's mental burden.

[1318] In this way, by combining generative AI and an emotion engine, the present invention provides a novel system that improves the efficiency of email processing and subsequent actions while taking into account the user's emotional state.

[1319] The processing flow will be explained below.

[1320] Step 1:

[1321] The server connects to the mail server using the IMAP protocol and retrieves the emails received that day from the specified mailbox (usually the inbox), which are then stored in a database.

[1322] Step 2:

[1323] The server extracts the text body of each retrieved email, which includes determining whether the email is multi-part or single-part and extracting the text from the appropriate part.

[1324] Step 3:

[1325] The server then uses generative AI to analyze the extracted email text and automatically generate a summary of the email's contents, which concisely summarizes the main points of the email.

[1326] Step 4:

[1327] The server determines the importance of an email based on metadata such as the sender and subject line of the email. For example, an email from your boss or about an important project might be considered high importance.

[1328] Step 5:

[1329] The server combines the generated summary with the determined importance to generate summary data, which includes information about the sender, subject, summary, and importance of the email.

[1330] Step 6:

[1331] The device uses a camera and microphone to analyze the user's facial expressions and tone of voice in real time to recognize the user's emotional state, thereby determining the user's stress level and emotional state.

[1332] Step 7:

[1333] The server receives the user's emotional state data and re-evaluates the importance of the emails based on the data, and determines the order in which the emails are processed that is optimal for the user's current emotional state.

[1334] Step 8:

[1335] The server references past response history and uses predictive AI to predict the next action to be taken for the generated summary. Based on this prediction, a specific action list is created.

[1336] Step 9:

[1337] The server generates a customized action list taking into account the user's emotional state and sends it to the user's terminal along with summary data. Based on the emotional state, actions that help reduce stress and urgent emails are prioritized.

[1338] Step 10:

[1339] The terminal displays the summary data sent from the server and a customized action list, through which the user can check a summary of all emails, their importance, and the next action to be taken.

[1340] Step 11:

[1341] The user can take necessary actions based on the summary and action list displayed on the device, such as sending a progress report in response to an important email from a boss, or performing a relaxation action when stress levels are high.

[1342] Through this series of processes, the system takes into account the user's emotional state to improve the efficiency of email processing and subsequent actions, thereby reducing the user's mental burden.

[1343] Example 2

[1344] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1345] In today's work environment, there is a huge amount of electronic communication, and people need to be able to quickly extract important information from it and respond appropriately. However, this takes time and effort, and it is especially difficult to respond effectively when users are stressed or tired. Furthermore, many current systems do not take users' emotional state into account and can only provide fixed responses, which hinders users' efficient work performance.

[1346] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for analyzing electronic communications using a generation AI and automatically generating a summary of the contents, means for determining the importance of the electronic communications, means for displaying the summary based on the determined importance, means for predicting the next action to be taken by referring to past response history and creating a work instruction list, means for displaying the work instruction list and summary on a user terminal, means for monitoring the user's emotional state, reevaluating the importance based on that information, and customizing the work instruction list, and means for analyzing emotional data in real time and feeding it back to the server. This enables efficient information extraction and response that takes the user's emotional state into consideration.

[1347] "Generative AI" is a technology that uses artificial intelligence to generate and analyze data.

[1348] "Electronic communications" means digital information sent or received over the Internet or an intranet.

[1349] A summary is a concise summary of the important points of a longer piece of text or data.

[1350] "Importance" is a measure of the priority or importance associated with particular information or data.

[1351] "Past response history" refers to records of responses and processes that have been taken in the past.

[1352] A "work instruction list" is a list of actions and procedures required to carry out a specific job or task.

[1353] A "user terminal" is a device that a user directly operates to display and input information. Generally, this includes PCs and smartphones.

[1354] "Emotional state" is a term that refers to a user's psychological state or mood.

[1355] "Feedback" refers to information or data given regarding the behavior or results of a system.

[1356] This invention relates to a system that uses generative AI to analyze electronic communications and automatically generate summaries of their contents, and also incorporates an emotion engine that recognizes the user's emotions, enabling the reevaluation of the importance of communications and the customization of action instruction lists.

[1357] Server Operation

[1358] The server first connects to the designated mail server to receive the electronic communication, which is accomplished using standard protocols such as IMAP (Internet Message Access Protocol) or POP3 (Post Office Protocol 3).

[1359] Examples:

[1360] The server connects to a communication server using the IMAP protocol and obtains new communications.

[1361] The captured communications are stored in an internal database.

[1362] The server then uses a generative AI model (such as OpenAI's GPT-4) to summarize the content of the received communication, designing an appropriate prompt and passing it to the generative AI.

[1363] Example prompt sentence:

[1364] "Please summarize the contents of the following electronic communication: 'Details about the progress of the project...'"

[1365] Based on the generated summary, the server uses an algorithm to determine the importance of the communication, based on factors such as past response history and the frequency of occurrence of fixed keywords.

[1366] Furthermore, the server uses the prediction function of the generative AI to predict the next action to be taken and create a list of action instructions. For example, for a communication about "checking project progress," which is a high-priority item, it suggests a specific action such as "submit a progress report."

[1367] Device behavior

[1368] The user device displays the summary and action instruction list sent from the server in real time, and the device's built-in emotion engine analyzes the user's facial expressions and tone of voice via a camera and microphone to recognize their emotional state.

[1369] Examples:

[1370] While the user is checking the communication, the device's camera captures the user's facial expressions, and the emotion engine performs emotion analysis in real time.

[1371] The microphone analyzes the tone of the user's voice and extracts data on the level of stress and excitement.

[1372] The extracted emotion data is fed back to the server, which then uses this data to reevaluate the importance of the communication and the action instruction list, customizing them as necessary.

[1373] How users use it

[1374] The user starts the system at the end of work and checks a summary of the day's received communications on their device. For example, a communication from a superior requesting a "project progress check" is displayed in the summary as being of high importance. If the user's emotional state indicates "stress," the system will prioritize actions to help reduce stress, such as "relax before starting to write a progress report," in the list of action instructions.

[1375] Hardware and software used

[1376] The following hardware and software are used to implement this system:

[1377] Server: Server machine that handles communication (e.g. Linux server)

[1378] Generative AI models: such as OpenAI's GPT-4

[1379] Emotion engine: Software with the ability to analyze a user's facial expressions and voice (e.g., Microsoft Azure's Face API or Speech API)

[1380] User device: PC or smartphone equipped with a camera and microphone

[1381] Through the above steps, the present invention realizes a series of processes that analyzes electronic communications, generates summaries, and provides action instructions that take emotional states into account. By using this system, users can process information and respond to business tasks efficiently while taking their emotions into consideration.

[1382] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1383] Step 1:

[1384] The server receives new electronic communications from a designated mail server.

[1385] Specific behavior:

[1386] The server connects to the mail server using the IMAP or POP3 protocol.

[1387] Obtain a list of new electronic communications and store their details in an internal database.

[1388] Input: mail server connection information, query for new electronic communications.

[1389] Output: A list of retrieved new electronic communications.

[1390] Step 2:

[1391] The server uses the generative AI model to summarize the content of the received electronic communication.

[1392] Specific behavior:

[1393] The server passes the following prompt to the generative AI model:

[1394] "Please summarize the contents of the following electronic communication: 'Details about the progress of the project...'"

[1395] The generative AI model returns a summary and stores the results in an internal database.

[1396] Input: Content of received electronic communication, prompt text.

[1397] Output: The generated summary.

[1398] Step 3:

[1399] The server determines the importance of the email.

[1400] Specific behavior:

[1401] The server uses an importance determination algorithm to rate the importance of the received electronic communication.

[1402] The importance score is stored in an internal database.

[1403] Input: Received electronic communications, summaries, and past correspondence history.

[1404] Output: Importance score.

[1405] Step 4:

[1406] The server predicts what action to take next and creates an action list.

[1407] Specific behavior:

[1408] The server passes the following prompt to the generative AI model:

[1409] "Suggest next actions for high priority emails"

[1410] The generative AI model suggests specific actions and stores the results as an action list in an internal database.

[1411] Input: Importance score, prompt for the generation AI.

[1412] Output: Action list.

[1413] Step 5:

[1414] The device monitors the user's emotional state and provides feedback to the server.

[1415] Specific behavior:

[1416] The camera captures the user's facial expressions, and the emotion engine analyzes them.

[1417] The microphone analyzes the tone of the user's voice and extracts emotional data.

[1418] The emotion data is sent to the server.

[1419] Input: User's facial expression data, tone of voice data.

[1420] Output: Emotion data.

[1421] Step 6:

[1422] The server reevaluates and customizes the importance and action list of emails based on emotional data.

[1423] Specific behavior:

[1424] The server receives the emotion data and evaluates the importance and action list again.

[1425] If necessary, add customizations to the action list, such as "Relax and then continue working."

[1426] Save the updated action list to an internal database.

[1427] Input: sentiment data, original action list, importance score.

[1428] Output: The updated action list.

[1429] Step 7:

[1430] The terminal displays a summary and a customized action list to the user.

[1431] Specific behavior:

[1432] The device retrieves the summary and latest action list from the server.

[1433] Display a summary and action list on the user interface.

[1434] Input: Summary, action list from server.

[1435] Output: The summary and action list that is displayed to the user.

[1436] The above is the specific processing flow of this system. Each processing step works together to achieve efficient analysis and response of electronic communications, taking into account the user's emotional state.

[1437] (Application example 2)

[1438] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1439] In conventional electronic payment services, users had to manually check and respond to all notifications and transaction emails, which was cumbersome and often mentally taxing. Furthermore, because the system did not take into account the user's emotional state, there were issues such as overlooking important transactions in stressful situations or being unable to make appropriate decisions regarding high-risk transactions.

[1440] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing emails using a generation AI and automatically generating a summary of the contents, means for determining the importance of the email, means for displaying the summary based on the determined importance, means for predicting the next action to be taken by referring to past response history and creating an action list, means for displaying the action list and the summary on a user terminal, means for analyzing the user's emotional state using a camera and microphone, and means for reevaluating the importance based on the user's emotional state and customizing the action list. This enables the user to make optimal decisions taking into consideration their emotional state and efficiently process important notifications and transactions related to electronic payments.

[1441] "Generative AI" refers to technology that uses artificial intelligence models to automatically generate text and data.

[1442] "Email" is a means of communication for sending and receiving text information, files, etc. via the Internet.

[1443] The "summary" is information that briefly summarizes the contents of the email.

[1444] "Importance" is a metric used to assess the urgency and priority of an email or notification.

[1445] "Past response history" refers to a record of responses to emails that a user has made up to now.

[1446] An "action list" is a list of tasks that indicate the next action the user should take.

[1447] "User terminal" is a general term for electronic devices used by users, including smartphones and tablets.

[1448] A "camera" is a device that captures video or still images.

[1449] A "microphone" is a device that records sound.

[1450] An "emotional state" is an emotion a user is feeling at a particular point in time, and may include "stressed" or "relaxed."

[1451] "Analysis" is a method of examining data and information in detail and understanding its content.

[1452] "Reevaluation" means to repeat or review an evaluation that has already been made.

[1453] "Customization" refers to adjusting specific content or settings to suit the user's needs and circumstances.

[1454] This invention is a system that uses generative AI to analyze emails and automatically generate summaries of their contents, and also incorporates an emotion engine that recognizes the user's emotions, enabling the re-evaluation of email importance and customization of action lists. This system consists of the following components:

[1455] Server Operation

[1456] The server connects to the specified mail server, retrieves, analyzes, and summarizes emails, and determines their importance based on their content. It also references past response history to predict the next action to be taken and uses generative AI to create an action list. The server analyzes the user's emotional state through a camera or microphone, and based on that information, reevaluates the importance of emails and customizes the action list. Examples of generative AI models used include GPT-3 and BERT.

[1457] User terminal operation

[1458] The user's device displays the summary and action list sent from the server in real time. The device is equipped with an emotion engine that uses a camera and microphone to analyze the user's facial expressions and tone of voice to recognize their emotional state. This emotional state is fed back to the server in real time, enabling the device to respond in a way that takes the user's emotions into consideration.

[1459] Specific examples

[1460] For example, when a user launches the application at the end of work and checks the summary of the day's received emails on their device, an email from their boss asking them to "check the project progress" is displayed in the summary as being of high importance. If the user's emotional state indicates "stress," the system will prioritize actions that help reduce stress, such as "relax before starting to write a progress report," in the action list.

[1461] Hardware and software used

[1462] Camera: Used to capture the user's facial expressions.

[1463] Microphone: Used to analyze the tone of the user's voice.

[1464] Python standard library: used to retrieve and parse emails (smtplib, imaplib, email).

[1465] TextBlob: Used to summarize the email content.

[1466] Emotion Recognition: A library for analyzing user emotions.

[1467] Prompt Sentence Examples

[1468] Summarize emails about "payment notifications" and "important transaction confirmations," determine their importance based on the user's emotional state, and generate appropriate action suggestions. If the user's emotion is determined to be "stressed," suggest that they relax before proceeding.

[1469] This allows users to make optimal decisions while taking their emotions into consideration, and efficiently take appropriate actions regarding electronic payments and mail processing.

[1470] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1471] Step 1:

[1472] The server connects to the specified mail server and retrieves emails. It requires mail server login information as input and provides the retrieved email data as output.

[1473] Step 2:

[1474] The server analyzes the retrieved email and summarizes its contents. The email body is required as input, and a generative AI model (e.g., TextBlob) is used to generate a summary, resulting in a summary text as output.

[1475] Step 3:

[1476] The server determines the importance of an email. It takes parsed email metadata (sender, subject, etc.) and summary text as input, evaluates the importance using an importance determination algorithm, and outputs an importance score.

[1477] Step 4:

[1478] The server references past correspondence history and predicts the next action to be taken. Emails and past correspondence history data are required as input, and a generative AI model is used to make the prediction, resulting in a list of actions as output.

[1479] Step 5:

[1480] The server transmits the action list and summary to the user terminal. The generated action list and summary text are required as input, and data transmission to the user terminal is required as output.

[1481] Step 6:

[1482] The terminal displays the summary and action list sent from the server in real time. It requires data from the server as input and provides visual information to the user as output.

[1483] Step 7:

[1484] The device analyzes the user's emotional state using a camera and microphone. This process requires the user's facial expression and voice data as input, analyzes the emotional state using an emotion recognition algorithm, and obtains emotional data as output.

[1485] Step 8:

[1486] The server receives the user's emotional data, re-evaluates the importance of emails based on the data, and customizes the action list. The server requires the emotional data and the existing action list as input, adjusts the action list using a re-evaluation algorithm, and obtains the customized action list as output.

[1487] Step 9:

[1488] Again, the server sends the customized action list to the user terminal and suggests appropriate actions to the user. A customized action list is required as input, and the final action list is provided to the user as output.

[1489] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1490] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1491] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1492] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1493] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1494] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1495] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1496] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1497] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1498] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1499] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1500] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1501] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1502] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1503] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1504] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1505] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1506] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1507] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1508] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1509] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1510] The following is further disclosed regarding the above embodiment.

[1511] (Claim 1)

[1512] A method for analyzing emails and automatically generating summaries of their contents using generative AI;

[1513] means for determining the importance of an email;

[1514] means for displaying a summary based on the determined importance;

[1515] A way to refer to past response history, predict what actions to take next, and create an action list.

[1516] A means for displaying the action list and summary on the user's terminal

[1517] A system including:

[1518] (Claim 2)

[1519] 10. The system of claim 1, further comprising means for retrieving received emails.

[1520] (Claim 3)

[1521] 2. The system according to claim 1, further comprising means for determining importance using metadata such as the sender and subject of the email.

[1522] "Example 1"

[1523] (Claim 1)

[1524] A method for analyzing emails and automatically generating summaries of their contents using generative AI;

[1525] means for determining the importance of an email;

[1526] means for displaying a summary based on the determined importance;

[1527] A way to refer to past response history, predict what actions to take next, and create an action list.

[1528] means for displaying the action list and summary on a user terminal;

[1529] means for connecting to a mail server using the IMAP protocol and retrieving received emails;

[1530] A means for inputting a prompt sentence into a generative AI model and analyzing the email using natural language processing;

[1531] A system including:

[1532] (Claim 2)

[1533] 10. The system of claim 1, further comprising means for storing received emails in a database.

[1534] (Claim 3)

[1535] 2. The system according to claim 1, further comprising means for determining importance using metadata such as the sender and subject of the email.

[1536] "Application Example 1"

[1537] (Claim 1)

[1538] A means for analyzing electronic communications and automatically generating summaries of their contents using generative AI;

[1539] a means for determining the importance of electronic communications;

[1540] means for displaying a summary based on the determined importance;

[1541] A means to refer to past response history, predict the next action to be taken, and create a response list;

[1542] means for displaying the correspondence list and summary on a user terminal;

[1543] a means of obtaining new electronic communications;

[1544] A means to automatically generate the next action to be taken from the summarized content and support customer responses,

[1545] A means for summarizing and predicting behavior using natural language processing with generative AI;

[1546] A system including:

[1547] (Claim 2)

[1548] 10. The system of claim 1, further comprising: means for obtaining new electronic communications.

[1549] (Claim 3)

[1550] 10. The system of claim 1, further comprising means for determining importance using metadata such as the sender and subject of the electronic communication.

[1551] "Example 2: Combining Emotion Engines"

[1552] (Claim 1)

[1553] A means for analyzing electronic communications and automatically generating summaries of their contents using generative AI;

[1554] a means for determining the importance of electronic communications;

[1555] means for displaying a summary based on the determined importance;

[1556] A means to refer to past response history, predict the next action to be taken, and create a work instruction list;

[1557] means for displaying the work instruction list and summary on a user terminal;

[1558] A means for monitoring the user's emotional state and reassessing the importance of tasks based on that information to customize the task list;

[1559] A means of analyzing emotion data in real time and feeding it back to the server

[1560] A system including:

[1561] (Claim 2)

[1562] 10. The system of claim 1, further comprising: means for capturing received electronic communications.

[1563] (Claim 3)

[1564] 10. The system of claim 1, further comprising means for determining importance using metadata such as the sender and subject of the electronic communication.

[1565] "Application example 2 when combining emotion engines"

[1566] (Claim 1)

[1567] A method for analyzing emails and automatically generating summaries of their contents using generative AI;

[1568] means for determining the importance of an email;

[1569] means for displaying a summary based on the determined importance;

[1570] A way to refer to past response history, predict what actions to take next, and create an action list.

[1571] means for displaying the action list and summary on a user terminal;

[1572] A means for analyzing the user's emotional state using a camera or microphone;

[1573] A way to reassess importance and customize action lists based on the user's emotional state

[1574] A system including:

[1575] (Claim 2)

[1576] 10. The system of claim 1, further comprising means for retrieving received emails.

[1577] (Claim 3)

[1578] 2. The system according to claim 1, further comprising means for determining importance using metadata such as the sender and subject of the email. [Explanation of symbols]

[1579] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A method for analyzing emails and automatically generating summaries of their contents using generative AI; means for determining the importance of an email; means for displaying a summary based on the determined importance; A way to refer to past response history, predict what actions to take next, and create an action list. A means for displaying the action list and summary on the user's terminal A system including:

2. The system of claim 1 further comprising means for capturing received emails.

3. The system according to claim 1, further comprising a means for determining importance using metadata such as the sender and subject of the email.

Images