system

Abstract

The system aims to reduce the time users spend checking and replying to messages, enabling more efficient communication. [Solution] A means for collecting messages from multiple communication services via a communication network; a means for analyzing the collected messages using natural language processing technology and generating summaries; a means for scoring the importance of the analyzed messages and sorting the messages based on importance; a means for sending automatically generated reply content to the user terminal using the user's past reply data; and a means for the user to modify the automatically generated reply content displayed on the user terminal and send the final reply. A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern society, communication through emails and various chat applications has become daily, but it has become a problem that a lot of time is spent on checking and replying to these messages. In particular, it is difficult to quickly identify important messages from a large amount of messages and reply appropriately, which is a factor causing a decrease in work efficiency. For this reason, there is a demand for a system aimed at reducing the time that users spend on checking and replying to messages and communicating more efficiently.

Means for Solving the Problems

[0005] The present invention solves the above problems with a system that includes means for collecting messages from multiple communication services via a communication network, means for analyzing the collected messages using natural language processing technology and generating summaries, means for scoring the importance of the analyzed messages and sorting the messages based on their importance, means for sending automatically generated replies to the user's terminal using the user's past reply data, and means for the user to make corrections to the automatically generated replies displayed on the user's terminal and send a final reply. As a result, users can significantly reduce the time spent checking and replying to messages and concentrate on important communication.

[0006] A "communication network" is an infrastructure for communicating data between multiple devices, such as the internet or an intranet.

[0007] "Communication services" refer to services that allow users to send and receive messages to each other, such as email, chat applications, and social media.

[0008] A "message" is a unit of information that is sent and received through communication services, including text, images, audio, and video.

[0009] "Means of collection" refers to the function of retrieving messages from communication services using an API.

[0010] "Natural language processing technology" refers to technologies for analyzing and understanding human language using computers, and includes tokenization, part-of-speech tagging, and semantic analysis of sentences.

[0011] "Means of analysis" refers to a function that uses natural language processing technology to analyze the content of a message and generate a summary.

[0012] A "summary" is information that extracts the key points of a message and presents them in a concise form.

[0013] A "means of scoring importance" refers to a function for numerically evaluating the importance and urgency of a message.

[0014] "Scoring" is the process of quantifying the importance of a message based on specific criteria.

[0015] The "sorting method" is a function that lists scored messages in descending order of their score.

[0016] "User's past reply data" refers to data about the content and patterns of messages that the user has previously sent.

[0017] "Automatically generated reply content" refers to a response message automatically created by the system based on the user's past reply data and common reply patterns.

[0018] A "terminal" is a device used by a user to access a system, and includes smartphones, tablets, personal computers, and other similar devices.

[0019] The "means of making corrections" refer to a function that allows users to review automatically generated replies and modify them as needed.

[0020] "Means of sending" refers to the function that allows a user to send a confirmed reply message through a communication service.

[0021] The "system" encompasses all of the above means and is a set of functions that enable users to efficiently check and reply to messages. [Brief explanation of the drawing]

[0022] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0023] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described according to the accompanying drawings.

[0024] First, the language used in the following description will be explained.

[0025] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0026] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0027] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0028] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0030] [First Embodiment]

[0031] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0032] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0035] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0038] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0042] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0043] This invention provides a system that automatically collects messages from multiple communication services used by a user, summarizes and evaluates their importance using natural language processing technology, and then provides the user with automatically generated replies. This system reduces the time and effort users spend checking and replying to messages.

[0044] System Configuration

[0045] This system includes the following main components:

[0046] 1. Collection method: The server collects messages from multiple communication services (e.g., email services, chat applications, social media) via a communication network.

[0047] 2. Analysis method: The server analyzes the collected messages using natural language processing technology and generates a summary.

[0048] 3. Scoring method: The server scores the importance of the analyzed messages and sorts them based on importance.

[0049] 4. Automatic generation method: The server uses the user's past reply data to automatically generate a reply and send it to the user's terminal.

[0050] 5. Correction and Submission Method: The user reviews the automatically generated reply displayed on their device, makes corrections as needed, and then submits the final reply.

[0051] Program processing

[0052] Collection methods

[0053] The server retrieves unread messages via APIs from Gmail, chat applications, and other services through the communication network. The server uses authentication tokens from each service to access user accounts and collect new and unread messages. The collected messages are stored in a database, which also stores metadata such as sender, recipient, and timestamp, in addition to the message content.

[0054] Analysis means

[0055] The server applies a natural language processing engine to the collected messages. This engine tokenizes the messages, tags them by part of speech, and analyzes the meaning of the sentences. As a result, a summary is generated and important information is extracted.

[0056] Scoring methods

[0057] The server uses a machine learning model to score the importance of the collected messages. This scoring takes into account factors such as the frequency of important keywords, the sender's role and relationship, and the urgency of past interactions. Based on the importance score, messages are sorted in order of importance.

[0058] automatic generation means

[0059] The server learns from the user's past reply data and creates automatically generated replies suitable for similar situations. These replies are sent to the user's device in JSON format, and the user can review them as drafts.

[0060] Correction / Transmission Methods

[0061] The terminal displays the automatically generated reply received from the server to the user. The user reviews the reply using the terminal's interface, makes any necessary corrections, and then confirms the final reply. The confirmed reply is sent to the server and then actually transmitted through the respective communication services.

[0062] Specific example

[0063] For example, if a user receives many unread emails or chat messages in a day, this system will work as follows:

[0064] 1. Collection method: The server uses the Gmail API and the chat app API to collect all unread messages.

[0065] 2. Analysis method: An NLP engine is applied to the collected messages to generate a summary.

[0066] 3. Scoring method: An importance score is calculated, and important messages are displayed at the top of the list.

[0067] 4. Automatic generation method: The server automatically generates a reply based on the user's past reply data and sends it to the user's terminal.

[0068] 5. Correction and Submission Method: The user reviews the reply on their device, makes any necessary corrections, confirms the message, and sends it.

[0069] This system significantly improves user work efficiency and reduces the time required to check and reply to messages.

[0070] The following describes the processing flow.

[0071] Step 1:

[0072] The server collects messages from communication services via the communication network. Specifically, it calls APIs of Gmail and chat applications to retrieve new and unread messages. In doing so, it uses API authentication tokens from each service to access the user's account.

[0073] Step 2:

[0074] The server stores the collected messages in a database. When saving, not only the message content but also metadata such as the sender, recipient, and timestamp are saved together.

[0075] Step 3:

[0076] The server initiates natural language processing on the stored messages. First, it tokenizes the messages, splitting them into individual words. Next, it tags each word with its part of speech and analyzes the meaning of the entire sentence. Based on this analysis, it generates a summary of the message.

[0077] Step 4:

[0078] The server uses a machine learning model to score importance based on the generated summary and message metadata. The scoring takes into account factors such as the frequency of important keywords, the relationship between the sender and receiver, and the urgency determined from past interactions.

[0079] Step 5:

[0080] The server sorts the scored messages by importance. Messages with higher scores are placed higher in the list, creating a list where important messages are displayed preferentially.

[0081] Step 6:

[0082] The server uses a model that has learned from the user's past reply data to create automatically generated replies. These replies are generated based on patterns in the user's past interactions and common reply templates.

[0083] Step 7:

[0084] The server sends the generated reply in JSON format to the user's device. The device receives this reply and displays it to the user.

[0085] Step 8:

[0086] The user reviews the automatically generated reply displayed on their device. They make any necessary corrections and finalize the reply.

[0087] Step 9:

[0088] The user submits their finalized reply by clicking the send button. The server then sends this reply through the API of the respective communication service (e.g., Gmail or chat application).

[0089] Through these steps, the system significantly reduces the time users spend checking and replying to messages, enabling more efficient communication.

[0090] (Example 1)

[0091] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0092] Currently, many users receive a large volume of messages daily through multiple communication services, resulting in the problem of significant time and effort being required to review and respond to them. Furthermore, the risk of overlooking important messages increases, highlighting the need for efficient message management. To address this challenge, a system is needed that can effectively automate message collection, summarization, importance assessment, and automatic responses.

[0093] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0094] In this invention, the server includes means for collecting messages from multiple communication services via a communication network; means for analyzing the collected messages using natural language processing technology and generating summaries; means for scoring the importance of the analyzed messages and sorting the messages based on importance; means for sending automatically generated reply content using the user's past reply data to the user terminal; means for the user to modify the automatically generated reply content displayed on the user terminal and send the final reply; means for storing metadata such as the sender, recipient, timestamp, and content of the collected messages; means for using a machine learning model to generate importance scores; means for tokenizing, tagging parts of speech, and analyzing sentences on the collected messages to extract important information; means for sending the automatically generated reply content in JSON format to the user terminal; and means for saving the reply content after user input and actually sending it via the communication service. As a result, users can significantly reduce the time required to check and reply to messages and efficiently manage important messages.

[0095] A "communication network" is the infrastructure used to send and receive data, and includes the internet and internal networks.

[0096] "Multiple communication services" refers to messaging services provided by different providers or applications, including, for example, email, chat apps, and social media.

[0097] A "message" is a unit of information that takes the form of text, images, audio, video, etc., and is sent from a sender to a recipient via a communication service.

[0098] "Natural language processing technology" refers to technologies for understanding, interpreting, and generating human language using computers, and includes processes such as tokenization, part-of-speech tagging, and semantic analysis.

[0099] A "summary" is information that extracts the most important information from the original message and presents it in a concise format.

[0100] "Importance" is a numerical value or rank that evaluates how important the content of a message is to the user.

[0101] "Scoring" is the process of evaluating the importance of a message and quantifying or ranking it.

[0102] "User's past reply data" refers to the history of reply messages previously sent by the user, and serves as reference data for creating automatically generated new reply content.

[0103] "Automatically generated reply content" refers to the content of a reply message that the system automatically creates based on the user's past reply data and other criteria.

[0104] A "user terminal" refers to a device such as a computer or smartphone used by a user, which can be used to check and reply to messages.

[0105] "Metadata" refers to information about the message itself, including attributes such as the sender, recipient, timestamp, and content.

[0106] A "machine learning model" is an algorithm or statistical model that uses large amounts of data to learn patterns and perform predictions or classifications.

[0107] "Tokenization" is the process of dividing text into meaningful units (tokens).

[0108] "Part-of-speech tagging" is the process of assigning a part of speech (noun, verb, adjective, etc.) to each token.

[0109] "Sentence analysis" is the process of analyzing the grammatical and semantic structure of a text.

[0110] JSON format is a lightweight data exchange format that represents data as key-value pairs.

[0111] This invention provides a system for efficiently managing messages by automatically collecting messages from multiple communication services used daily by users, and analyzing and organizing them using natural language processing technology. This system consists of a server and a user terminal and includes the following means.

[0112] System hardware and software

[0113] server

[0114] The server has the following functions:

[0115] A function that collects messages via a communication network. Specific examples include using the Gmail API or chat application APIs.

[0116] A function to store collected messages in a database. This uses a database management system such as MySQL (registered trademark).

[0117] A function that applies natural language processing technology. It can utilize Google's BERT® algorithm.

[0118] A function that generates a summary based on the analysis results.

[0119] A feature that uses machine learning models to score importance and sort messages accordingly.

[0120] This feature automatically generates replies based on the user's past reply data. It utilizes machine learning libraries such as TENSORFLOW® and PyTorch for this purpose.

[0121] A function that sends automatically generated reply content to the user's terminal in JSON format.

[0122] User terminal

[0123] The user terminal has the following functions:

[0124] A function that displays automatically generated replies received from the server to the user.

[0125] A web application is used as an interface for users to review replies and make corrections as needed.

[0126] A function that sends the revised reply to the server and then finally sends the message through each communication service.

[0127] Data processing and data calculation

[0128] The server collects messages using the APIs of various communication services and stores them in a database. Metadata such as sender, recipient, timestamp, and content is also stored. Next, a natural language processing engine (such as BERT) is used to analyze the messages, extract important information, and generate summaries. Furthermore, a machine learning model is used to score the importance of the messages and sort them accordingly.

[0129] Based on the user's past reply data, an automatically generated reply is created and sent to the user's terminal in JSON format. The user reviews the content displayed on their terminal, makes any necessary corrections, and sends the final message to the server. The server then sends this message to the respective communication services, and the message is actually sent.

[0130] Specific example

[0131] For example, if a user receives many unread emails in their Gmail account, the server uses the Gmail API to collect these messages and store them in a database. A natural language processing engine is applied to the collected messages to generate summaries. Next, a machine learning model is used to calculate importance scores and sort the messages so that the most important ones are displayed first. The automatically generated reply is sent to the user's terminal in JSON format, and after the user reviews and modifies it, the final reply is sent to the communication service.

[0132] Example of a prompt

[0133] Examples of prompt statements to input into a generative AI model include:

[0134] "Collect unread messages received by users, generate summaries, assess their importance, and create prompts to generate automated replies."

[0135] The above describes the "modes for carrying out the invention" of the system related to the present invention. This system allows users to significantly reduce the time and effort required to check and reply to messages.

[0136] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0137] Step 1: Collecting Messages

[0138] Server operation:

[0139] Input: Use user authentication information (such as an OAuth token) to send requests to the APIs of various communication services (Gmail, chat apps, social media, etc.).

[0140] Data processing: Receive unread message data obtained from communication services and convert it to the required format.

[0141] Output: Content and metadata of unread messages (sender, recipient, timestamp, etc.).

[0142] Specific operation: The server sends a request to the Gmail API, retrieves unread messages, and stores them in the "messages" table in the database.

[0143] Step 2: Message Analysis

[0144] Server operation:

[0145] Input: Content and metadata of unread messages stored in the database.

[0146] Data processing: Apply a natural language processing engine (such as BERT) to tokenize messages, tag them by part of speech, and perform semantic analysis of the sentences.

[0147] Output: Message summary and extracted key information.

[0148] Specific operation: The server analyzes the message "Today's meeting is at 3 PM" and generates a summary such as "Meeting change."

[0149] Step 3: Importance scoring

[0150] Server operation:

[0151] Input: Summary and metadata of the analyzed message (sender's job title and relationship, urgency level, etc.).

[0152] Data processing: An importance score is calculated using a machine learning model. Scoring factors include keyword frequency and the urgency of past interactions.

[0153] Output: Importance score and a list of messages sorted by importance.

[0154] Specific operation: The server evaluates emails from superiors regarding "meeting changes" with a high score and places them higher in the list.

[0155] Step 4: Automatically generate reply content

[0156] Server operation:

[0157] Input: User's past reply data and a list of messages sorted by importance.

[0158] Data processing: Generate appropriate response content based on past response data. This is done using machine learning libraries (e.g., TensorFlow or PyTorch).

[0159] Output: Auto-generated reply content in JSON format.

[0160] Specific operation: The server generates a new "Understood" message based on the user's previous "Understood" reply and sends it to the user's terminal in JSON format.

[0161] Step 5: Presenting the response to the user

[0162] Device operation:

[0163] Input: Automated response content in JSON format received from the server.

[0164] Data processing: Convert the received JSON data into a user-friendly format and display it on the interface.

[0165] Output: The reply displayed to the user.

[0166] Specific action: The device displays an automatically generated reply message to the user saying "Understood."

[0167] Step 6: User modification and submission

[0168] User actions:

[0169] Input: The automatically generated reply content that was displayed.

[0170] Data processing: Modify the reply content as needed.

[0171] Output: The revised and final reply.

[0172] Specific action: The user changes "Understood" to "Understood. I have also confirmed the meeting time," and then presses the send button.

[0173] Step 7: Send the revised reply.

[0174] Server operation:

[0175] Input: The corrected reply received from the user's terminal.

[0176] Data processing: The modified response content is sent via the API of each communication service.

[0177] Output: The final reply message sent to each communication service.

[0178] Specific action: The server actually sends the corrected reply, "Understood. I've also confirmed the meeting time," via the Gmail API.

[0179] (Application Example 1)

[0180] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0181] In modern logistics centers, numerous employees exchange information using a variety of communication methods, resulting in a massive volume of messages, which poses a significant challenge. In this environment, quickly identifying and responding appropriately to important messages is crucial, but doing so manually is time-consuming and labor-intensive, hindering efficient operations. Furthermore, while automated generation of replies based on relevant past response data is desirable, effective methods for achieving this have lacked.

[0182] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0183] In this invention, the server includes means for collecting messages from multiple communication services via a communication network; means for analyzing the collected messages using natural language processing technology and generating summaries; means for scoring the importance of the analyzed messages and sorting the messages based on importance; means for sending automatically generated replies to a user terminal using the user's past reply data; means for the user to modify the automatically generated replies displayed on the user terminal and send the final reply; and means for automatically collecting and analyzing messages from multiple communication tools used by employees in the logistics center and automatically generating appropriate replies. This makes it possible to efficiently manage a vast number of messages in the logistics center, respond quickly to important messages, and improve work efficiency by automatically generating appropriate replies.

[0184] A "communication network" is the infrastructure used to connect computers to each other and to send and receive data.

[0185] "Communication services" refer to platforms for sending and receiving digital messages, such as email, chat applications, and social media.

[0186] "Message" refers to a unit of text, images, audio, and other data transmitted via electronic means of communication.

[0187] Natural language processing is a branch of computer science that analyzes and understands human language, and is often used for tasks such as text summarization and sentiment analysis.

[0188] A "summary" is a format in which the content of the original message is shortened and only the important information is extracted.

[0189] "Scoring" refers to the process of numerically evaluating the importance of a message, which makes it possible to prioritize messages.

[0190] A "user terminal" is a device used by a user to operate something, and includes smartphones, tablets, and personal computers.

[0191] "Reply data" refers to a history of messages previously sent by the user.

[0192] "Automatic generation" refers to the process of creating messages and replies based on pre-programmed algorithms, without human intervention.

[0193] A "logistics center" is a facility for storing, sorting, and shipping goods, and its purpose is to enable efficient logistics management.

[0194] "Employees" refers to workers who work at the logistics center and are responsible for tasks such as information gathering, sorting, shipping, and management.

[0195] "Communication tools" refer to applications and software used to send and receive messages, such as email, chat applications, and notifications from IoT devices.

[0196] This invention is a system for achieving efficient message management and rapid response within a logistics center. The system configuration and specific embodiments are described below.

[0197] System Configuration

[0198] The system includes the following main components:

[0199] 1. Server: Collects messages from multiple communication services via the communication network, and analyzes, summarizes, scores, and filters them using natural language processing technology. It also creates automatically generated replies based on the user's past reply data and sends them to the user's terminal.

[0200] 2. User terminal: Used by employees at the logistics center to view summaries of messages sent from the server, automatically generated replies, make corrections, and send final replies.

[0201] 3. Communication Network: This is the infrastructure that connects servers and user terminals and allows the use of multiple communication service APIs.

[0202] Hardware and software configuration

[0203] hardware

[0204] User devices: Smartphones, tablets, personal computers, etc.

[0205] Server: Responsible for data processing as the central management system.

[0206] software

[0207] Python: Used for implementing natural language processing and machine learning models.

[0208] Generative AI models: GPT-3 (registered trademark), BERT, etc., used for text generation and information extraction.

[0209] APIs from various communication services, such as the Gmail API, Slack API, and IoT device APIs, are used to collect messages.

[0210] Database: Used for managing messages and metadata.

[0211] Processing flow

[0212] The server first collects new and unread messages through the APIs of each communication service. Next, it applies a natural language processing engine (e.g., SpaCy) to the collected messages to generate summaries. Then, it uses a machine learning model (e.g., importance scoring using TF-IDF) to score the importance of the messages and sorts them in order of importance.

[0213] Subsequently, the server automatically generates appropriate replies using a generative AI model (e.g., GPT-3) based on the user's past reply data. This generated reply is sent to the user's terminal in JSON format, allowing the user to review, revise, and submit their final reply as a draft.

[0214] Specific example

[0215] For example, if an employee at a logistics center receives a large number of unread messages and notifications, this system will work as follows:

[0216] 1. Message Collection and Analysis: The server uses the Gmail API, Slack API, and IoT device APIs to collect all unread messages and notifications, and generates summaries using a natural language processing engine.

[0217] 2. Scoring and Sorting: Use a machine learning model to calculate importance scores and prioritize and sort important messages.

[0218] 3. Automatic generation of replies: The server uses an AI model to automatically generate appropriate replies based on the collected messages.

[0219] 4. Review and correction on the user's device: The user reviews the reply on their device, makes any necessary corrections, and then confirms and sends the final reply.

[0220] Examples of prompts for generative AI models

[0221] example:

[0222] Please generate a response appropriate to the following situation:

[0223] situation:

[0224] We received an inquiry from a customer stating that their ordered item has not yet arrived. Upon checking the tracking information, we found that the delivery is delayed.

[0225] reply:

[0226] This system streamlines message management within the logistics center, significantly improving employee work efficiency. Furthermore, it enables quick responses to important messages, contributing to increased customer satisfaction.

[0227] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0228] Step 1:

[0229] The server collects messages from multiple communication services (such as Gmail, Slack, and IoT devices) via a communication network. In this step, it uses APIs to retrieve new and unread messages and saves them to a local database. The input is the message data obtained from each communication service, and the output is the message data stored in the local database.

[0230] Step 2:

[0231] The server applies natural language processing techniques to the collected messages to generate summaries. In this step, libraries such as SpaCy are used to tokenize the messages, extract important information, and generate summaries. The input is message data obtained from a local database, and the output is the generated summary data. Specifically, the first few sentences of the message body are extracted and used as the summary.

[0232] Step 3:

[0233] The server scores and sorts the importance of the analyzed messages. In this step, a machine learning model (e.g., TF-IDF scoring) is used to evaluate the importance of the messages. The input is summarized message data, and the output is a list of messages that have been scored and sorted by importance. Specifically, the TF-IDF score for each message is calculated, and the messages are sorted based on that score.

[0234] Step 4:

[0235] The server uses the user's past reply data to create automatically generated replies using a generative AI model (e.g., GPT-3). In this step, the reply is generated based on the prompt. The input is scored message data and the user's past reply data, and the output is the generated reply. Specifically, the prompt is input to the generative AI model, and the generated reply is retrieved.

[0236] Step 5:

[0237] The server automatically generates a reply and sends it to the user's terminal. In this step, the reply generated in JSON format is transferred to the user's terminal. The input is the generated reply, and the output is a notification that the transmission to the user's terminal is complete. Specifically, the reply is sent using the user's terminal's identification information.

[0238] Step 6:

[0239] The user terminal displays the automatically generated reply to the user. In this step, the reply is displayed on the interface, and the user is provided with the ability to make modifications. The input is the automatically generated reply sent from the server, and the output is the reply that has been reviewed and modified by the user. Specifically, the reply is rendered on the display interface.

[0240] Step 7:

[0241] The user makes final revisions to their reply and sends it. In this step, the user reviews and confirms the revised reply, and then sends it via the server through the respective communication services. The input is the reply revised by the user, and the output is the final reply message sent. Specifically, based on the user's confirmation action, the message is sent through the respective communication service APIs.

[0242] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0243] This invention is a system that automatically collects messages from multiple communication services used by a user, analyzes them using natural language processing technology and an emotion engine to summarize and evaluate their importance, and provides the user with automatically generated replies. This system reduces the time and effort users spend checking and replying to messages, and also enables appropriate responses that take into account the user's emotions.

[0244] System Configuration

[0245] This system includes the following main components:

[0246] 1. Collection method: The server collects messages from multiple communication services (email services, chat applications, social media) via a communication network.

[0247] 2. Analysis method: The server analyzes the collected messages using natural language processing technology and generates a summary.

[0248] 3. Scoring method: The server scores the importance of the analyzed messages and sorts them based on importance.

[0249] 4. Emotion Recognition Method: The server uses an emotion engine to recognize emotions from collected messages.

[0250] 5. Automatic generation means: The server automatically generates a reply based on the user's past reply data and recognized emotions and sends it to the user's terminal.

[0251] 6. Correction and Submission Method: The user reviews the automatically generated reply displayed on their device, makes corrections as needed, and then submits the final reply.

[0252] Program processing

[0253] Collection methods

[0254] The server retrieves unread messages via APIs from services such as Gmail and chat applications through the communication network. It accesses user accounts using API authentication tokens for each service to collect new and unread messages. The collected messages are stored in a database, which stores not only the message content but also metadata such as sender, recipient, and timestamp.

[0255] Analysis means

[0256] The server applies a natural language processing engine to the collected messages. First, the messages are tokenized and split into individual words. Next, the part of speech of each word is identified, the meaning of the sentence is analyzed, and a message summary is generated.

[0257] emotion recognition means

[0258] The server uses an emotion engine to recognize the sentiment of a message. The emotion engine extracts emotions from the text within the message and assigns sentiment labels such as positive, negative, and neutral. The recognized sentiment influences importance scoring and automated reply generation.

[0259] Scoring methods

[0260] The server uses a machine learning model to score the importance of the collected messages. The scoring considers factors such as the frequency of important keywords, the sender's role and relationship, and the results of sentiment recognition. Based on the importance score, messages are sorted in order of importance.

[0261] automatic generation means

[0262] The server automatically generates a response based on the user's past reply data and perceived emotions. This response is generated based on patterns of past interactions and common reply templates. In particular, it generates a response with an appropriate tone and content that takes the results of emotion recognition into account.

[0263] Correction / Transmission Methods

[0264] The device displays the automatically generated reply received from the server to the user. The user reviews the reply on the device, makes any necessary corrections, and then confirms the final reply. The confirmed reply is sent to the server and then actually transmitted through the APIs of each communication service (e.g., Gmail or chat applications).

[0265] Specific example

[0266] For example, if a user receives many unread emails or chat messages in a day, this system will work as follows:

[0267] 1. Collection method: The server uses the Gmail API and the chat app API to collect all unread messages.

[0268] 2. Analysis method: An NLP engine is applied to the collected messages to generate a summary.

[0269] 3. Emotion Recognition Method: The emotion engine recognizes the emotion of the message and assigns an emotion label.

[0270] 4. Scoring method: An importance score is calculated, and important messages are displayed at the top of the list.

[0271] 5. Automatic generation method: The server automatically generates a reply based on the user's past reply data and sentiment recognition results, and sends it to the user's terminal.

[0272] 6. Correction and Submission Method: The user reviews the reply on their device, makes any necessary corrections, confirms the message, and sends it.

[0273] This system significantly improves user productivity, reducing the time required to check and reply to messages, and enabling appropriate responses that take user emotions into consideration.

[0274] The following describes the processing flow.

[0275] Step 1:

[0276] The server collects messages from communication services via the communication network. Specifically, it calls APIs of Gmail and chat applications to retrieve new and unread messages. In doing so, it uses API authentication tokens from each service to access the user's account.

[0277] Step 2:

[0278] The server saves the messages collected into a database. When saving, not only the content of the messages but also metadata such as the sender, recipient, and timestamp are saved together.

[0279] Step 3:

[0280] The server starts natural language processing (NLP) on the saved messages. First, the messages are tokenized and split into individual words. Next, each word is tagged with its part of speech, and the overall meaning of the sentence is analyzed to generate a summary of the message.

[0281] Step 4:

[0282] The server uses an emotion engine to recognize the emotion of the messages. The emotion engine extracts emotion labels such as positive, negative, and neutral from the messages and adds this emotion information to the message data.

[0283] Step 5:

[0284] The server uses a machine learning model to score the importance of the collected messages. For scoring, factors such as the frequency of occurrence of important keywords, the sender's position and relationships, and the results of emotion recognition are considered. Based on the importance scores, the messages are sorted in order of importance.

[0285] Step 6:

[0286] The server creates automatically generated reply content based on the user's past reply data and the results of emotion recognition. The generated reply content is based on past interaction patterns and general reply templates, and includes replies with an appropriate tone and content that particularly consider the results of emotion recognition.

[0287] Step 7:

[0288] The server sends the generated reply in JSON format to the user's device. The device receives this reply and displays it to the user.

[0289] Step 8:

[0290] The user reviews the automatically generated reply displayed on their device. They make any necessary corrections and finalize the reply.

[0291] Step 9:

[0292] The user submits their finalized reply by clicking the send button. The server then sends this reply through the API of the respective communication service (e.g., Gmail or chat application).

[0293] Through this series of steps, users can significantly reduce the time spent checking and responding to messages, and also be able to provide appropriate responses that take into account the user's emotions.

[0294] (Example 2)

[0295] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0296] In modern society, users utilize multiple communication services (email, chat applications, social media, etc.) and receive a large volume of messages daily. However, responding quickly and appropriately to all messages is difficult, and managing and replying to messages consumes a significant amount of time and effort. Furthermore, accurately understanding the content, importance, and sentiment of a message, and responding accordingly, requires high skill. Therefore, there is a need for technology that can quickly generate appropriate responses and reduce the user's time and effort.

[0297] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting messages from multiple communication services via a communication network, means for storing metadata (sender, recipient, timestamp, content) of the collected messages, means for tokenizing the collected messages using natural language processing technology, assigning part-of-speech tags, and generating summaries, means for recognizing the sentiment of messages using a sentiment engine and labeling the results, means for scoring the importance of messages analyzed using a machine learning model and sorting messages based on importance, means for sending automatically generated reply content based on the user's past reply data and recognized sentiment to the user terminal, and means for the user to modify the automatically generated reply content displayed on the user terminal and send the final reply. This enables the user to respond quickly and appropriately to a large volume of messages, as well as significantly reduce the user's time and effort.

[0298] A "communication network" refers to a computer network used for sending and receiving data.

[0299] "Communication services" refer to services that allow users to exchange information, such as email, chat applications, and social media.

[0300] A "message" refers to information such as text, images, and files sent by a user.

[0301] "Metadata" refers to supplementary information accompanying a message, including attribute data such as sender, recipient, timestamp, and content.

[0302] "Natural language processing technology" refers to the technology used to process natural language used by humans using computers.

[0303] "Tokenization" refers to the process of dividing a continuous text into words or phrases.

[0304] "Part-of-speech tagging" refers to the process of identifying which part of speech each tokenized word belongs to, such as noun, verb, adjective, etc.

[0305] "Summarization" refers to the text that extracts the main points of the original message and summarizes them concisely.

[0306] "Sentiment engine" refers to an algorithm for recognizing and labeling sentiment from text data.

[0307] "Machine learning model" refers to an algorithm that learns patterns from data and performs analysis and prediction of new data based on the learning results.

[0308] "Importance scoring" refers to the process of numerically evaluating the importance of a message.

[0309] "Sorting" refers to the process of ordering messages based on importance scores.

[0310] "Automatically generated reply content" refers to the reply message created by a computer based on the user's past reply patterns and recognized sentiment.

[0311] "User terminal" refers to the device (such as smartphone, tablet, personal computer, etc.) for the user to access the system.

[0312] "Modification" refers to the act of the user making changes or supplements to the automatically generated reply content.

[0313] "Final reply" refers to the reply message determined after the user makes modifications.

[0314] This invention is a system that automatically collects messages from multiple communication services used by a user, analyzes them using natural language processing technology and an emotion engine to perform summarization and importance evaluation, and provides the user with automatically generated replies.

[0315] This system includes the following main components:

[0316] 1. Collection Method: The server collects messages from multiple communication services (email services, chat applications, social media) via the communication network. Specifically, it uses the Gmail API and chat application APIs to retrieve users' unread messages. It accesses user accounts using API authentication tokens for each service to collect new and unread messages. The collected messages are stored in a database, and in addition to the message content, metadata such as sender, recipient, and timestamp is also stored.

[0317] 2. Analysis Method: The server tokenizes the collected messages using natural language processing techniques (e.g., SpaCy, NLTK) and tags them by part of speech. Tokenization divides the message text into words, and part-of-speech tagging identifies the part of speech of each word. Next, the server performs semantic analysis of the sentences and generates a summary of the messages. The NLTK summarization algorithm is used for this purpose.

[0318] 3. Emotion Recognition Method: The server uses an emotion engine (e.g., Hugging Face's Transformers model) to recognize the emotion of a message. The emotion engine extracts emotions from the text within the message and assigns emotion labels such as positive, negative, and neutral. The recognized emotion influences importance scoring and automatic reply generation.

[0319] 4. Scoring Method: The server uses a machine learning model (e.g., a classification model from Scikit-learn) to score the importance of the collected messages. The scoring considers factors such as the frequency of important keywords, the sender's role and relationship, and the results of sentiment recognition. Based on the importance score, messages are sorted in order of importance.

[0320] 5. Automatic generation method: The server automatically generates response content based on the user's past response data and recognized emotions. This response content is generated based on patterns of past interactions and general response templates. In particular, responses with appropriate tone and content are generated, taking into account the results of emotion recognition. The server generates responses using a generation AI model (e.g., GPT-3).

[0321] 6. Correction and Sending Method: The terminal displays the automatically generated reply received from the server to the user. The user reviews the reply on the terminal, makes any necessary corrections, and then confirms the final reply. The confirmed reply is sent to the server and actually sent through the APIs of each communication service (e.g., Gmail or chat applications).

[0322] As a concrete example, if a user receives many unread emails and chat messages in a day, this system will work as follows:

[0323] 1. The server uses the Gmail API and the chat app API to collect all unread messages.

[0324] 2. A natural language processing engine is applied to the collected messages to generate a summary.

[0325] 3. The emotion engine recognizes the emotion of the message and assigns an emotion label.

[0326] 4. An importance score is calculated, and important messages are displayed at the top of the list.

[0327] 5. The server automatically generates a reply based on the user's past reply data and sentiment recognition results, and sends it to the user's terminal.

[0328] 6. The user reviews the reply on their device, makes any necessary corrections, confirms it, and sends it.

[0329] Examples of input prompts for a generative AI model are shown below:

[0330] 1. "Please describe the process for automatically collecting unread messages from email and chat applications, generating summaries, sorting them based on importance scores, and automatically generating appropriate replies."

[0331] 2. "Explain how to use an emotion recognition engine to label the emotions of messages and then use that information to automatically generate replies."

[0332] These prompts allow the user to get guidance on the detailed operation and implementation of the system.

[0333] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0334] Program processing flow

[0335] Step 1:

[0336] The server collects unread messages from multiple communication services (e.g., email services, chat applications, social media) via a communication network. The input requires API authentication tokens and user IDs for each communication service. The server uses these to call each API and retrieve new and unread messages. The output includes metadata such as message content, sender, recipient, and timestamp, which are stored in a database.

[0337] Specific actions:

[0338] The server calls the Gmail API to retrieve the user's unread emails.

[0339] The server uses the chat application API to collect unread messages.

[0340] Step 2:

[0341] The server uses a natural language processing engine (e.g., SpaCy) to tokenize the messages it collects. The message text is required as input. The server divides the message text into words and outputs a sequence of tokenized words.

[0342] Specific actions:

[0343] The server uses SpaCy's nlp(text) function to tokenize the message text.

[0344] Step 3:

[0345] The server performs part-of-speech tagging on tokenized words. The input is a sequence of tokenized words. The server assigns a part-of-speech tag to each word, and the output is a sequence of words tagged with part-of-speech.

[0346] Specific actions:

[0347] The part-of-speech information of the tokens generated by SpaCy is extracted, and tags are assigned to each token.

[0348] Step 4:

[0349] The server performs semantic analysis of the text and generates a message summary. The input requires a sequence of words tagged with parts of speech. The server uses natural language processing techniques (e.g., NLTK) to extract key points and obtains a summary as output.

[0350] Specific actions:

[0351] The server uses the NLTK summarization algorithm to extract important sentences.

[0352] Step 5:

[0353] The server uses an emotion engine (e.g., Hugging Face's Transformers model) to recognize emotions in the collected messages. The message text is required as input. The emotion engine parses the text and outputs emotion labels such as positive, negative, or neutral.

[0354] Specific actions:

[0355] The server calls pipeline('sentiment-analysis') to retrieve sentiment labels from the message text.

[0356] Step 6:

[0357] The server uses a machine learning model (e.g., a classification model in Scikit-learn) to score the importance of collected messages. Input includes sentiment labels, important keywords, and sender information. The server calculates a score based on this information, and the output is the importance score.

[0358] Specific actions:

[0359] We will use a classification model from Scikit-learn to calculate an importance score for each message.

[0360] Step 7:

[0361] The server sorts messages based on their importance score. The input is a list of scored messages. The server sorts the messages by importance and outputs a sorted list of messages.

[0362] Specific actions:

[0363] The server sorts the message list based on its importance score, placing the most important messages at the top of the list.

[0364] Step 8:

[0365] The server automatically generates response content based on the user's past response data and recognized sentiment. Past response data and sentiment labels are required as input. A generative AI model (e.g., GPT-3) is used to generate a response based on the prompt, and the response content is obtained as output.

[0366] Specific actions:

[0367] This function calls the GPT-3 API and generates a response based on the prompt.

[0368] Input prompt example: "Generate an appropriate reply for the unread message."

[0369] Step 9:

[0370] The terminal displays the automatically generated reply received from the server to the user. The generated reply is required as input. The user reviews the reply on the terminal, makes any necessary corrections, and then receives the final confirmed reply as output.

[0371] Specific actions:

[0372] The user reviews and modifies the reply displayed on their device.

[0373] Step 10:

[0374] The user sends their confirmed reply to the server, which then uses the APIs of the respective communication services to actually send the final reply. The input requires the user's confirmed and revised final reply. The server sends the reply via the API and notifies the user when the transmission is complete.

[0375] Specific actions:

[0376] The server calls the Gmail API to send the email.

[0377] Send a message using the chat application API.

[0378] (Application Example 2)

[0379] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0380] In modern brick-and-mortar stores, customer inquiries are increasingly pouring in through various communication services. This means that store employees have to spend a considerable amount of time and effort reviewing and responding to messages appropriately. Furthermore, failure to respond immediately can lead to decreased customer satisfaction. Therefore, there is a need for technology that can efficiently manage messages and automatically generate appropriate responses.

[0381] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting messages from multiple communication services via a communication network, means for analyzing the collected messages using natural language processing technology and generating summaries, means for scoring the importance of the analyzed messages and sorting the messages based on importance, means for sending automatically generated reply content using the user's past reply data to the user terminal, means for the user to modify the automatically generated reply content displayed on the user terminal and send the final reply, means for applying emotion recognition technology to the collected messages and assigning emotion labels, and means for generating automatic reply content to respond to customer inquiry messages in a physical store, which is then modified and sent by the store staff. This makes it possible to respond to customer inquiries quickly and appropriately.

[0382] A "communication network" is an infrastructure for exchanging information between different locations, such as the internet or mobile phone networks.

[0383] A "message" is a unit of information that is sent and received in various formats, such as text, images, and videos.

[0384] "Natural language processing technology" is a technology that uses computers to analyze and understand human language.

[0385] A "means of generating summaries" is a technique for concisely summarizing the main content and key points of collected messages.

[0386] "Scoring importance" is the process of quantifying the importance of a message based on the content and sender information collected from that message.

[0387] "Emotion recognition technology" is a technology that detects emotions from data such as text and audio, and assigns specific emotion labels (positive, negative, neutral, etc.) to them.

[0388] "Automatically generated replies" are replies that a computer automatically creates based on the user's past reply patterns and templates.

[0389] A "user terminal" is a device (such as a smartphone or personal computer) that a user uses to receive or send information.

[0390] "A means of making revisions and sending a final reply" refers to the process by which a user reviews an automatically generated reply, makes changes as needed, and then sends the message.

[0391] A "physical store" is a store that exists physically and is a place where commercial transactions are conducted directly with customers.

[0392] An "inquiry message" is a message sent by a customer to a store or service provider with questions or requests.

[0393] This invention provides a system that enables users to efficiently manage messages from multiple communication services and provide prompt and appropriate responses. This invention significantly reduces the time and effort required to check and reply to messages. Furthermore, it enables appropriate responses that take into account the user's emotions.

[0394] System Configuration

[0395] This system includes the following main components:

[0396] 1. Collection method: The server collects messages from multiple communication services (e.g., email, chat applications, social media) via a communication network. The collected messages are stored in a database.

[0397] 2. Analysis Method: The server analyzes the collected messages using natural language processing (NLP) techniques and generates summaries. An NLP engine such as the TextBlob library is used for natural language processing.

[0398] 3. Scoring method: The server uses a machine learning model (e.g., scikit-learn's logistic regression model) to score the importance of messages. Based on the importance score, messages are sorted in order of importance.

[0399] 4. Emotion Recognition Method: The server uses emotion recognition technology (e.g., TextBlob library) to recognize the emotions of the collected messages and assign emotion labels such as positive, negative, or neutral.

[0400] 5. Automatic generation method: The server uses the user's past reply data and sends automatically generated reply content based on the recognized sentiment to the user's terminal. This response content is generated based on templates or corpora.

[0401] 6. Modification and Submission Method: The user makes modifications to the automatically generated reply displayed on their device and then submits the final reply. The modified reply is sent via the API of the respective communication service (e.g., email or chat application).

[0402] Hardware and software configuration

[0403] Hardware: Smartphones, PCs, servers

[0404] software:

[0405] Collection method: Requests library

[0406] Analysis method: TextBlob library

[0407] Scoring method: scikit-learn library

[0408] Emotion recognition method: TextBlob library

[0409] Automatic generation method: Template engine (e.g., Jinja2)

[0410] Modification / Sending Method: smtplib library (for sending emails)

[0411] Specific example

[0412] For example, in a physical store, if a customer sends a message inquiring about product availability via various communication services, this system will operate as follows:

[0413] 1. The server collects all inquiry messages using APIs from services such as Instagram, LINE, and Facebook Messenger.

[0414] 2. Apply an NLP engine to the collected messages to generate a summary.

[0415] 3. Use emotion recognition technology to recognize the emotions in a message and assign emotion labels.

[0416] 4. An importance score is generated, and important messages are displayed at the top of the list.

[0417] 5. The server automatically generates a reply based on the user's past reply data and sentiment recognition results, and sends it to the user's terminal.

[0418] 6. The user reviews the reply on their device, makes any necessary corrections, confirms it, and sends it.

[0419] Example of a prompt

[0420] "Please create an appropriate template for automatically sending replies inquiring about product availability."

[0421] With the above configuration, the system of the present invention is capable of responding quickly and appropriately to customer inquiries. This is expected to improve customer satisfaction and the work efficiency of store employees.

[0422] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0423] Step 1:

[0424] The server collects messages from multiple communication services via a communication network. The server uses APIs to retrieve unread messages from services such as email, chat applications, and social media. The input is the API authentication token for each service, and the output is the retrieved message and its metadata (sender, recipient, timestamp, content, etc.). The collected messages are stored in a database.

[0425] Step 2:

[0426] This process applies natural language processing (NLP) techniques to messages collected by the server to generate summaries. An NLP engine (e.g., TextBlob library) is used to tokenize and parse the messages. The input is the collected message body, and the output is the generated summary. The process extracts key parts from the message to create a concise summary.

[0427] Step 3:

[0428] The server uses sentiment recognition technology to recognize the sentiment of the collected messages. It uses the TextBlob library to extract sentiment labels (positive, negative, neutral, etc.) from the text within the messages. The input is the summarized message body, and the output is a summary with sentiment labels added. Appropriate sentiment labels are added according to the tone of the message.

[0429] Step 4:

[0430] The server uses a machine learning model to score the importance of collected messages. It uses a scikit-learn logistic regression model to calculate importance scores based on input data (summaries, sentiment labels, and metadata). The input consists of sentiment-labeled summaries and metadata, and the output is the importance score. Based on the scoring results, the messages are sorted in order of importance.

[0431] Step 5:

[0432] The server uses the user's past reply data to generate automatically generated replies based on the perceived sentiment. A template engine (e.g., Jinja2) is used to create the automated replies. Inputs include importance scores, sorted messages, and the user's past reply patterns; output is the automatically generated reply. Contextually appropriate replies are generated based on templates.

[0433] Step 6:

[0434] The user reviews the automatically generated reply displayed on their device, makes any necessary corrections, and sends the final reply. The user checks the reply content using a smartphone or computer and makes corrections as needed. The input is the automatically generated reply, and the output is the corrected, final reply. After the user confirms and approves the reply, it is sent via the API of the respective communication service.

[0435] Through the processing steps described above, the server can efficiently execute a series of processes from message collection to automatic reply generation. For example, in response to a message inquiring about product inventory status, an automatic reply can be quickly generated, allowing store staff to modify and send it as needed.

[0436] Example prompt: "Please create an appropriate template for automatically sending a message inquiring about product availability."

[0437] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0438] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0439] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0440] [Second Embodiment]

[0441] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0442] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0443] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0444] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0445] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0446] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0447] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0448] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0449] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0450] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0451] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0452] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0453] This invention provides a system that automatically collects messages from multiple communication services used by a user, summarizes and evaluates their importance using natural language processing technology, and then provides the user with automatically generated replies. This system reduces the time and effort users spend checking and replying to messages.

[0454] System Configuration

[0455] This system includes the following main components:

[0456] 1. Collection method: The server collects messages from multiple communication services (e.g., email services, chat applications, social media) via a communication network.

[0457] 2. Analysis method: The server analyzes the collected messages using natural language processing technology and generates a summary.

[0458] 3. Scoring method: The server scores the importance of the analyzed messages and sorts them based on importance.

[0459] 4. Automatic generation method: The server uses the user's past reply data to automatically generate a reply and send it to the user's terminal.

[0460] 5. Correction and Submission Method: The user reviews the automatically generated reply displayed on their device, makes corrections as needed, and then submits the final reply.

[0461] Program processing

[0462] Collection methods

[0463] The server retrieves unread messages via APIs from Gmail, chat applications, and other services through the communication network. The server uses authentication tokens from each service to access user accounts and collect new and unread messages. The collected messages are stored in a database, which also stores metadata such as sender, recipient, and timestamp, in addition to the message content.

[0464] Analysis means

[0465] The server applies a natural language processing engine to the collected messages. This engine tokenizes the messages, tags them by part of speech, and analyzes the meaning of the sentences. As a result, a summary is generated and important information is extracted.

[0466] Scoring methods

[0467] The server uses a machine learning model to score the importance of the collected messages. This scoring takes into account factors such as the frequency of important keywords, the sender's role and relationship, and the urgency of past interactions. Based on the importance score, messages are sorted in order of importance.

[0468] automatic generation means

[0469] The server learns from the user's past reply data and creates automatically generated replies suitable for similar situations. These replies are sent to the user's device in JSON format, and the user can review them as drafts.

[0470] Correction / Transmission Methods

[0471] The terminal displays the automatically generated reply received from the server to the user. The user reviews the reply using the terminal's interface, makes any necessary corrections, and then confirms the final reply. The confirmed reply is sent to the server and then actually transmitted through the respective communication services.

[0472] Specific example

[0473] For example, if a user receives many unread emails or chat messages in a day, this system will work as follows:

[0474] 1. Collection method: The server uses the Gmail API and the chat app API to collect all unread messages.

[0475] 2. Analysis method: An NLP engine is applied to the collected messages to generate a summary.

[0476] 3. Scoring method: An importance score is calculated, and important messages are displayed at the top of the list.

[0477] 4. Automatic generation method: The server automatically generates a reply based on the user's past reply data and sends it to the user's terminal.

[0478] 5. Correction and Submission Method: The user reviews the reply on their device, makes any necessary corrections, confirms the message, and sends it.

[0479] This system significantly improves user work efficiency and reduces the time required to check and reply to messages.

[0480] The following describes the processing flow.

[0481] Step 1:

[0482] The server collects messages from communication services via the communication network. Specifically, it calls APIs of Gmail and chat applications to retrieve new and unread messages. In doing so, it uses API authentication tokens from each service to access the user's account.

[0483] Step 2:

[0484] The server stores the collected messages in a database. When saving, not only the message content but also metadata such as the sender, recipient, and timestamp are saved together.

[0485] Step 3:

[0486] The server initiates natural language processing on the stored messages. First, it tokenizes the messages, splitting them into individual words. Next, it tags each word with its part of speech and analyzes the meaning of the entire sentence. Based on this analysis, it generates a summary of the message.

[0487] Step 4:

[0488] The server uses a machine learning model to score importance based on the generated summary and message metadata. The scoring takes into account factors such as the frequency of important keywords, the relationship between the sender and receiver, and the urgency determined from past interactions.

[0489] Step 5:

[0490] The server sorts the scored messages by importance. Messages with higher scores are placed higher in the list, creating a list where important messages are displayed preferentially.

[0491] Step 6:

[0492] The server uses a model that has learned from the user's past reply data to create automatically generated replies. These replies are generated based on patterns in the user's past interactions and common reply templates.

[0493] Step 7:

[0494] The server sends the generated reply in JSON format to the user's device. The device receives this reply and displays it to the user.

[0495] Step 8:

[0496] The user reviews the automatically generated reply displayed on their device. They make any necessary corrections and finalize the reply.

[0497] Step 9:

[0498] The user submits their finalized reply by clicking the send button. The server then sends this reply through the API of the respective communication service (e.g., Gmail or chat application).

[0499] Through these steps, the system significantly reduces the time users spend checking and replying to messages, enabling more efficient communication.

[0500] (Example 1)

[0501] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0502] Currently, many users receive a large volume of messages daily through multiple communication services, resulting in the problem of significant time and effort being required to review and respond to them. Furthermore, the risk of overlooking important messages increases, highlighting the need for efficient message management. To address this challenge, a system is needed that can effectively automate message collection, summarization, importance assessment, and automatic responses.

[0503] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0504] In this invention, the server includes means for collecting messages from multiple communication services via a communication network; means for analyzing the collected messages using natural language processing technology and generating summaries; means for scoring the importance of the analyzed messages and sorting the messages based on importance; means for sending automatically generated reply content using the user's past reply data to the user terminal; means for the user to modify the automatically generated reply content displayed on the user terminal and send the final reply; means for storing metadata such as the sender, recipient, timestamp, and content of the collected messages; means for using a machine learning model to generate importance scores; means for tokenizing, tagging parts of speech, and analyzing sentences on the collected messages to extract important information; means for sending the automatically generated reply content in JSON format to the user terminal; and means for saving the reply content after user input and actually sending it via the communication service. As a result, users can significantly reduce the time required to check and reply to messages and efficiently manage important messages.

[0505] A "communication network" is the infrastructure used to send and receive data, and includes the internet and internal networks.

[0506] "Multiple communication services" refers to messaging services provided by different providers or applications, including, for example, email, chat apps, and social media.

[0507] A "message" is a unit of information that takes the form of text, images, audio, video, etc., and is sent from a sender to a recipient via a communication service.

[0508] "Natural language processing technology" refers to technologies for understanding, interpreting, and generating human language using computers, and includes processes such as tokenization, part-of-speech tagging, and semantic analysis.

[0509] A "summary" is information that extracts the most important information from the original message and presents it in a concise format.

[0510] "Importance" is a numerical value or rank that evaluates how important the content of a message is to the user.

[0511] "Scoring" is the process of evaluating the importance of a message and quantifying or ranking it.

[0512] "User's past reply data" refers to the history of reply messages previously sent by the user, and serves as reference data for creating automatically generated new reply content.

[0513] "Automatically generated reply content" refers to the content of a reply message that the system automatically creates based on the user's past reply data and other criteria.

[0514] A "user terminal" refers to a device such as a computer or smartphone used by a user, which can be used to check and reply to messages.

[0515] "Metadata" refers to information about the message itself, including attributes such as the sender, recipient, timestamp, and content.

[0516] A "machine learning model" is an algorithm or statistical model that uses large amounts of data to learn patterns and perform predictions or classifications.

[0517] "Tokenization" is the process of dividing text into meaningful units (tokens).

[0518] "Part-of-speech tagging" is the process of assigning a part of speech (noun, verb, adjective, etc.) to each token.

[0519] "Sentence analysis" is the process of analyzing the grammatical and semantic structure of a text.

[0520] JSON format is a lightweight data exchange format that represents data as key-value pairs.

[0521] This invention provides a system for efficiently managing messages by automatically collecting messages from multiple communication services used daily by users, and analyzing and organizing them using natural language processing technology. This system consists of a server and a user terminal and includes the following means.

[0522] System hardware and software

[0523] server

[0524] The server has the following functions:

[0525] A function that collects messages via a communication network. Specific examples include using the Gmail API or chat application APIs.

[0526] A function to store collected messages in a database. This uses a database management system such as MySQL.

[0527] A function that applies natural language processing technology. Google's BERT can be used.

[0528] A function that generates a summary based on the analysis results.

[0529] A feature that uses machine learning models to score importance and sort messages accordingly.

[0530] This feature automatically generates replies based on the user's past response data. It utilizes machine learning libraries such as TensorFlow and PyTorch for this purpose.

[0531] A function that sends automatically generated reply content to the user's terminal in JSON format.

[0532] User terminal

[0533] The user terminal has the following functions:

[0534] A function that displays automatically generated replies received from the server to the user.

[0535] A web application is used as an interface for users to review replies and make corrections as needed.

[0536] A function that sends the revised reply to the server and then finally sends the message through each communication service.

[0537] Data processing and data calculation

[0538] The server collects messages using the APIs of various communication services and stores them in a database. Metadata such as sender, recipient, timestamp, and content is also stored. Next, a natural language processing engine (such as BERT) is used to analyze the messages, extract important information, and generate summaries. Furthermore, a machine learning model is used to score the importance of the messages and sort them accordingly.

[0539] Based on the user's past reply data, an automatically generated reply is created and sent to the user's terminal in JSON format. The user reviews the content displayed on their terminal, makes any necessary corrections, and sends the final message to the server. The server then sends this message to the respective communication services, and the message is actually sent.

[0540] Specific example

[0541] For example, if a user receives many unread emails in their Gmail account, the server uses the Gmail API to collect these messages and store them in a database. A natural language processing engine is applied to the collected messages to generate summaries. Next, a machine learning model is used to calculate importance scores and sort the messages so that the most important ones are displayed first. The automatically generated reply is sent to the user's terminal in JSON format, and after the user reviews and modifies it, the final reply is sent to the communication service.

[0542] Example of a prompt

[0543] Examples of prompt statements to input into a generative AI model include:

[0544] "Collect unread messages received by users, generate summaries, assess their importance, and create prompts to generate automated replies."

[0545] The above describes the "modes for carrying out the invention" of the system related to the present invention. This system allows users to significantly reduce the time and effort required to check and reply to messages.

[0546] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0547] Step 1: Collecting Messages

[0548] Server operation:

[0549] Input: Use user authentication information (such as an OAuth token) to send requests to the APIs of various communication services (Gmail, chat apps, social media, etc.).

[0550] Data processing: Receive unread message data obtained from communication services and convert it to the required format.

[0551] Output: Content and metadata of unread messages (sender, recipient, timestamp, etc.).

[0552] Specific operation: The server sends a request to the Gmail API, retrieves unread messages, and stores them in the "messages" table in the database.

[0553] Step 2: Message Analysis

[0554] Server operation:

[0555] Input: Content and metadata of unread messages stored in the database.

[0556] Data processing: Apply a natural language processing engine (such as BERT) to tokenize messages, tag them by part of speech, and perform semantic analysis of the sentences.

[0557] Output: Message summary and extracted key information.

[0558] Specific operation: The server analyzes the message "Today's meeting is at 3 PM" and generates a summary such as "Meeting change."

[0559] Step 3: Importance scoring

[0560] Server operation:

[0561] Input: Summary and metadata of the analyzed message (sender's job title and relationship, urgency level, etc.).

[0562] Data processing: An importance score is calculated using a machine learning model. Scoring factors include keyword frequency and the urgency of past interactions.

[0563] Output: Importance score and a list of messages sorted by importance.

[0564] Specific operation: The server evaluates emails from superiors regarding "meeting changes" with a high score and places them higher in the list.

[0565] Step 4: Automatically generate reply content

[0566] Server operation:

[0567] Input: User's past reply data and a list of messages sorted by importance.

[0568] Data processing: Generate appropriate response content based on past response data. This is done using machine learning libraries (e.g., TensorFlow or PyTorch).

[0569] Output: Auto-generated reply content in JSON format.

[0570] Specific operation: The server generates a new "Understood" message based on the user's previous "Understood" reply and sends it to the user's terminal in JSON format.

[0571] Step 5: Presenting the response to the user

[0572] Device operation:

[0573] Input: Automated response content in JSON format received from the server.

[0574] Data processing: Convert the received JSON data into a user-friendly format and display it on the interface.

[0575] Output: The reply displayed to the user.

[0576] Specific action: The device displays an automatically generated reply message to the user saying "Understood."

[0577] Step 6: User modification and submission

[0578] User actions:

[0579] Input: The automatically generated reply content that was displayed.

[0580] Data processing: Modify the reply content as needed.

[0581] Output: The revised and final reply.

[0582] Specific action: The user changes "Understood" to "Understood. I have also confirmed the meeting time," and then presses the send button.

[0583] Step 7: Send the revised reply.

[0584] Server operation:

[0585] Input: The corrected reply received from the user's terminal.

[0586] Data processing: The modified response content is sent via the API of each communication service.

[0587] Output: The final reply message sent to each communication service.

[0588] Specific action: The server actually sends the corrected reply, "Understood. I've also confirmed the meeting time," via the Gmail API.

[0589] (Application Example 1)

[0590] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0591] In modern logistics centers, numerous employees exchange information using a variety of communication methods, resulting in a massive volume of messages, which poses a significant challenge. In this environment, quickly identifying and responding appropriately to important messages is crucial, but doing so manually is time-consuming and labor-intensive, hindering efficient operations. Furthermore, while automated generation of replies based on relevant past response data is desirable, effective methods for achieving this have lacked.

[0592] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0593] In this invention, the server includes means for collecting messages from multiple communication services via a communication network; means for analyzing the collected messages using natural language processing technology and generating summaries; means for scoring the importance of the analyzed messages and sorting the messages based on importance; means for sending automatically generated replies to a user terminal using the user's past reply data; means for the user to modify the automatically generated replies displayed on the user terminal and send the final reply; and means for automatically collecting and analyzing messages from multiple communication tools used by employees in the logistics center and automatically generating appropriate replies. This makes it possible to efficiently manage a vast number of messages in the logistics center, respond quickly to important messages, and improve work efficiency by automatically generating appropriate replies.

[0594] A "communication network" is the infrastructure used to connect computers to each other and to send and receive data.

[0595] "Communication services" refer to platforms for sending and receiving digital messages, such as email, chat applications, and social media.

[0596] "Message" refers to a unit of text, images, audio, and other data transmitted via electronic means of communication.

[0597] Natural language processing is a branch of computer science that analyzes and understands human language, and is often used for tasks such as text summarization and sentiment analysis.

[0598] A "summary" is a format in which the content of the original message is shortened and only the important information is extracted.

[0599] "Scoring" refers to the process of numerically evaluating the importance of a message, which makes it possible to prioritize messages.

[0600] A "user terminal" is a device used by a user to operate something, and includes smartphones, tablets, and personal computers.

[0601] "Reply data" refers to a history of messages previously sent by the user.

[0602] "Automatic generation" refers to the process of creating messages and replies based on pre-programmed algorithms, without human intervention.

[0603] A "logistics center" is a facility for storing, sorting, and shipping goods, and its purpose is to enable efficient logistics management.

[0604] "Employees" refers to workers who work at the logistics center and are responsible for tasks such as information gathering, sorting, shipping, and management.

[0605] "Communication tools" refer to applications and software used to send and receive messages, such as email, chat applications, and notifications from IoT devices.

[0606] This invention is a system for achieving efficient message management and rapid response within a logistics center. The system configuration and specific embodiments are described below.

[0607] System Configuration

[0608] The system includes the following main components:

[0609] 1. Server: Collects messages from multiple communication services via the communication network, and analyzes, summarizes, scores, and filters them using natural language processing technology. It also creates automatically generated replies based on the user's past reply data and sends them to the user's terminal.

[0610] 2. User terminal: Used by employees at the logistics center to view summaries of messages sent from the server, automatically generated replies, make corrections, and send final replies.

[0611] 3. Communication Network: This is the infrastructure that connects servers and user terminals and allows the use of multiple communication service APIs.

[0612] Hardware and software configuration

[0613] hardware

[0614] User devices: Smartphones, tablets, personal computers, etc.

[0615] Server: Responsible for data processing as the central management system.

[0616] software

[0617] Python: Used for implementing natural language processing and machine learning models.

[0618] Generative AI models: GPT-3, BERT, etc., used for text generation and information extraction.

[0619] APIs from various communication services, such as the Gmail API, Slack API, and IoT device APIs, are used to collect messages.

[0620] Database: Used for managing messages and metadata.

[0621] Processing flow

[0622] The server first collects new and unread messages through the APIs of each communication service. Next, it applies a natural language processing engine (e.g., SpaCy) to the collected messages to generate summaries. Then, it uses a machine learning model (e.g., importance scoring using TF-IDF) to score the importance of the messages and sorts them in order of importance.

[0623] Subsequently, the server automatically generates appropriate replies using a generative AI model (e.g., GPT-3) based on the user's past reply data. This generated reply is sent to the user's terminal in JSON format, allowing the user to review, revise, and submit their final reply as a draft.

[0624] Specific example

[0625] For example, if an employee at a logistics center receives a large number of unread messages and notifications, this system will work as follows:

[0626] 1. Message Collection and Analysis: The server uses the Gmail API, Slack API, and IoT device APIs to collect all unread messages and notifications, and generates summaries using a natural language processing engine.

[0627] 2. Scoring and Sorting: Use a machine learning model to calculate importance scores and prioritize and sort important messages.

[0628] 3. Automatic generation of replies: The server uses an AI model to automatically generate appropriate replies based on the collected messages.

[0629] 4. Review and correction on the user's device: The user reviews the reply on their device, makes any necessary corrections, and then confirms and sends the final reply.

[0630] Examples of prompts for generative AI models

[0631] example:

[0632] Please generate a response appropriate to the following situation:

[0633] situation:

[0634] We received an inquiry from a customer stating that their ordered item has not yet arrived. Upon checking the tracking information, we found that the delivery is delayed.

[0635] reply:

[0636] This system streamlines message management within the logistics center, significantly improving employee work efficiency. Furthermore, it enables quick responses to important messages, contributing to increased customer satisfaction.

[0637] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0638] Step 1:

[0639] The server collects messages from multiple communication services (such as Gmail, Slack, and IoT devices) via a communication network. In this step, it uses APIs to retrieve new and unread messages and saves them to a local database. The input is the message data obtained from each communication service, and the output is the message data stored in the local database.

[0640] Step 2:

[0641] The server applies natural language processing techniques to the collected messages to generate summaries. In this step, libraries such as SpaCy are used to tokenize the messages, extract important information, and generate summaries. The input is message data obtained from a local database, and the output is the generated summary data. Specifically, the first few sentences of the message body are extracted and used as the summary.

[0642] Step 3:

[0643] The server scores and sorts the importance of the analyzed messages. In this step, a machine learning model (e.g., TF-IDF scoring) is used to evaluate the importance of the messages. The input is summarized message data, and the output is a list of messages that have been scored and sorted by importance. Specifically, the TF-IDF score for each message is calculated, and the messages are sorted based on that score.

[0644] Step 4:

[0645] The server uses the user's past reply data to create automatically generated replies using a generative AI model (e.g., GPT-3). In this step, the reply is generated based on the prompt. The input is scored message data and the user's past reply data, and the output is the generated reply. Specifically, the prompt is input to the generative AI model, and the generated reply is retrieved.

[0646] Step 5:

[0647] The server automatically generates a reply and sends it to the user's terminal. In this step, the reply generated in JSON format is transferred to the user's terminal. The input is the generated reply, and the output is a notification that the transmission to the user's terminal is complete. Specifically, the reply is sent using the user's terminal's identification information.

[0648] Step 6:

[0649] The user terminal displays the automatically generated reply to the user. In this step, the reply is displayed on the interface, and the user is provided with the ability to make modifications. The input is the automatically generated reply sent from the server, and the output is the reply that has been reviewed and modified by the user. Specifically, the reply is rendered on the display interface.

[0650] Step 7:

[0651] The user makes final revisions to their reply and sends it. In this step, the user reviews and confirms the revised reply, and then sends it via the server through the respective communication services. The input is the reply revised by the user, and the output is the final reply message sent. Specifically, based on the user's confirmation action, the message is sent through the respective communication service APIs.

[0652] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0653] This invention is a system that automatically collects messages from multiple communication services used by a user, analyzes them using natural language processing technology and an emotion engine to summarize and evaluate their importance, and provides the user with automatically generated replies. This system reduces the time and effort users spend checking and replying to messages, and also enables appropriate responses that take into account the user's emotions.

[0654] System Configuration

[0655] This system includes the following main components:

[0656] 1. Collection method: The server collects messages from multiple communication services (email services, chat applications, social media) via a communication network.

[0657] 2. Analysis method: The server analyzes the collected messages using natural language processing technology and generates a summary.

[0658] 3. Scoring method: The server scores the importance of the analyzed messages and sorts them based on importance.

[0659] 4. Emotion Recognition Method: The server uses an emotion engine to recognize emotions from collected messages.

[0660] 5. Automatic generation means: The server automatically generates a reply based on the user's past reply data and recognized emotions and sends it to the user's terminal.

[0661] 6. Correction and Submission Method: The user reviews the automatically generated reply displayed on their device, makes corrections as needed, and then submits the final reply.

[0662] Program processing

[0663] Collection methods

[0664] The server retrieves unread messages via APIs from services such as Gmail and chat applications through the communication network. It accesses user accounts using API authentication tokens for each service to collect new and unread messages. The collected messages are stored in a database, which stores not only the message content but also metadata such as sender, recipient, and timestamp.

[0665] Analysis means

[0666] The server applies a natural language processing engine to the collected messages. First, the messages are tokenized and split into individual words. Next, the part of speech of each word is identified, the meaning of the sentence is analyzed, and a message summary is generated.

[0667] emotion recognition means

[0668] The server uses an emotion engine to recognize the sentiment of a message. The emotion engine extracts emotions from the text within the message and assigns sentiment labels such as positive, negative, and neutral. The recognized sentiment influences importance scoring and automated reply generation.

[0669] Scoring methods

[0670] The server uses a machine learning model to score the importance of the collected messages. The scoring considers factors such as the frequency of important keywords, the sender's role and relationship, and the results of sentiment recognition. Based on the importance score, messages are sorted in order of importance.

[0671] automatic generation means

[0672] The server automatically generates a response based on the user's past reply data and perceived emotions. This response is generated based on patterns of past interactions and common reply templates. In particular, it generates a response with an appropriate tone and content that takes the results of emotion recognition into account.

[0673] Correction / Transmission Methods

[0674] The device displays the automatically generated reply received from the server to the user. The user reviews the reply on the device, makes any necessary corrections, and then confirms the final reply. The confirmed reply is sent to the server and then actually transmitted through the APIs of each communication service (e.g., Gmail or chat applications).

[0675] Specific example

[0676] For example, if a user receives many unread emails or chat messages in a day, this system will work as follows:

[0677] 1. Collection method: The server uses the Gmail API and the chat app API to collect all unread messages.

[0678] 2. Analysis method: An NLP engine is applied to the collected messages to generate a summary.

[0679] 3. Emotion Recognition Method: The emotion engine recognizes the emotion of the message and assigns an emotion label.

[0680] 4. Scoring method: An importance score is calculated, and important messages are displayed at the top of the list.

[0681] 5. Automatic generation method: The server automatically generates a reply based on the user's past reply data and sentiment recognition results, and sends it to the user's terminal.

[0682] 6. Correction and Submission Method: The user reviews the reply on their device, makes any necessary corrections, confirms the message, and sends it.

[0683] This system significantly improves user productivity, reducing the time required to check and reply to messages, and enabling appropriate responses that take user emotions into consideration.

[0684] The following describes the processing flow.

[0685] Step 1:

[0686] The server collects messages from communication services via the communication network. Specifically, it calls APIs of Gmail and chat applications to retrieve new and unread messages. In doing so, it uses API authentication tokens from each service to access the user's account.

[0687] Step 2:

[0688] The server stores the collected messages in a database. When saving, not only the message content but also metadata such as the sender, recipient, and timestamp are saved together.

[0689] Step 3:

[0690] The server initiates natural language processing (NLP) on the stored messages. First, the messages are tokenized and split into individual words. Next, part-of-speech tags are added to each word, and the meaning of the entire sentence is analyzed to generate a summary of the message.

[0691] Step 4:

[0692] The server uses an emotion engine to recognize the emotion of a message. The emotion engine extracts emotion labels such as positive, negative, and neutral from the message and adds this emotion information to the message data.

[0693] Step 5:

[0694] The server uses a machine learning model to score the importance of the collected messages. The scoring considers factors such as the frequency of important keywords, the sender's role and relationship, and the results of sentiment recognition. Based on the importance score, messages are sorted in order of importance.

[0695] Step 6:

[0696] The server automatically generates replies based on the user's past reply data and sentiment recognition results. The generated replies are based on patterns of past interactions and common reply templates, and include replies with an appropriate tone and content, taking into account the sentiment recognition results in particular.

[0697] Step 7:

[0698] The server sends the generated reply in JSON format to the user's device. The device receives this reply and displays it to the user.

[0699] Step 8:

[0700] The user reviews the automatically generated reply displayed on their device. They make any necessary corrections and finalize the reply.

[0701] Step 9:

[0702] The user submits their finalized reply by clicking the send button. The server then sends this reply through the API of the respective communication service (e.g., Gmail or chat application).

[0703] Through this series of steps, users can significantly reduce the time spent checking and responding to messages, and also be able to provide appropriate responses that take into account the user's emotions.

[0704] (Example 2)

[0705] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0706] In modern society, users utilize multiple communication services (email, chat applications, social media, etc.) and receive a large volume of messages daily. However, responding quickly and appropriately to all messages is difficult, and managing and replying to messages consumes a significant amount of time and effort. Furthermore, accurately understanding the content, importance, and sentiment of a message, and responding accordingly, requires high skill. Therefore, there is a need for technology that can quickly generate appropriate responses and reduce the user's time and effort.

[0707] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting messages from multiple communication services via a communication network, means for storing metadata (sender, recipient, timestamp, content) of the collected messages, means for tokenizing the collected messages using natural language processing technology, assigning part-of-speech tags, and generating summaries, means for recognizing the sentiment of messages using a sentiment engine and labeling the results, means for scoring the importance of messages analyzed using a machine learning model and sorting messages based on importance, means for sending automatically generated reply content based on the user's past reply data and recognized sentiment to the user terminal, and means for the user to modify the automatically generated reply content displayed on the user terminal and send the final reply. This enables the user to respond quickly and appropriately to a large volume of messages, as well as significantly reduce the user's time and effort.

[0708] A "communication network" refers to a computer network used for sending and receiving data.

[0709] "Communication services" refer to services that allow users to exchange information, such as email, chat applications, and social media.

[0710] A "message" refers to information such as text, images, and files sent by a user.

[0711] "Metadata" refers to supplementary information accompanying a message, including attribute data such as sender, recipient, timestamp, and content.

[0712] "Natural language processing technology" refers to the technology used to process natural language used by humans using computers.

[0713] "Tokenization" refers to the process of dividing a continuous text into words or phrases.

[0714] "Part-of-speech tagging" refers to the process of identifying which part of speech each tokenized word belongs to, such as a noun, verb, or adjective.

[0715] A "summary" refers to a text that extracts the main points of the original message and presents them concisely.

[0716] An "emotion engine" refers to an algorithm that recognizes and labels emotions from text data.

[0717] A "machine learning model" refers to an algorithm that learns patterns from data and uses those learning results to analyze and predict new data.

[0718] "Importance scoring" refers to the process of numerically evaluating the importance of a message.

[0719] "Sorting" refers to the process of ordering messages based on their importance score.

[0720] "Automatically generated replies" refer to reply messages created by a computer based on the user's past reply patterns and perceived emotions.

[0721] "User terminal" refers to a device (such as a smartphone, tablet, or personal computer) that a user uses to access the system.

[0722] "Correction" refers to the act of a user making changes or additions to automatically generated replies.

[0723] "Final reply" refers to the reply message that the user has finalized after making revisions.

[0724] This invention is a system that automatically collects messages from multiple communication services used by a user, analyzes them using natural language processing technology and an emotion engine to perform summarization and importance evaluation, and provides the user with automatically generated replies.

[0725] This system includes the following main components:

[0726] 1. Collection Method: The server collects messages from multiple communication services (email services, chat applications, social media) via the communication network. Specifically, it uses the Gmail API and chat application APIs to retrieve users' unread messages. It accesses user accounts using API authentication tokens for each service to collect new and unread messages. The collected messages are stored in a database, and in addition to the message content, metadata such as sender, recipient, and timestamp is also stored.

[0727] 2. Analysis Method: The server tokenizes the collected messages using natural language processing techniques (e.g., SpaCy, NLTK) and tags them by part of speech. Tokenization divides the message text into words, and part-of-speech tagging identifies the part of speech of each word. Next, the server performs semantic analysis of the sentences and generates a summary of the messages. The NLTK summarization algorithm is used for this purpose.

[0728] 3. Emotion Recognition Method: The server uses an emotion engine (e.g., Hugging Face's Transformers model) to recognize the emotion of a message. The emotion engine extracts emotions from the text within the message and assigns emotion labels such as positive, negative, and neutral. The recognized emotion influences importance scoring and automatic reply generation.

[0729] 4. Scoring Method: The server uses a machine learning model (e.g., a classification model from Scikit-learn) to score the importance of the collected messages. The scoring considers factors such as the frequency of important keywords, the sender's role and relationship, and the results of sentiment recognition. Based on the importance score, messages are sorted in order of importance.

[0730] 5. Automatic generation method: The server automatically generates response content based on the user's past response data and recognized emotions. This response content is generated based on patterns of past interactions and general response templates. In particular, responses with appropriate tone and content are generated, taking into account the results of emotion recognition. The server generates responses using a generation AI model (e.g., GPT-3).

[0731] 6. Correction and Sending Method: The terminal displays the automatically generated reply received from the server to the user. The user reviews the reply on the terminal, makes any necessary corrections, and then confirms the final reply. The confirmed reply is sent to the server and actually sent through the APIs of each communication service (e.g., Gmail or chat applications).

[0732] As a concrete example, if a user receives many unread emails and chat messages in a day, this system will work as follows:

[0733] 1. The server uses the Gmail API and the chat app API to collect all unread messages.

[0734] 2. A natural language processing engine is applied to the collected messages to generate a summary.

[0735] 3. The emotion engine recognizes the emotion of the message and assigns an emotion label.

[0736] 4. An importance score is calculated, and important messages are displayed at the top of the list.

[0737] 5. The server automatically generates a reply based on the user's past reply data and sentiment recognition results, and sends it to the user's terminal.

[0738] 6. The user reviews the reply on their device, makes any necessary corrections, confirms it, and sends it.

[0739] Examples of input prompts for a generative AI model are shown below:

[0740] 1. "Please describe the process for automatically collecting unread messages from email and chat applications, generating summaries, sorting them based on importance scores, and automatically generating appropriate replies."

[0741] 2. "Explain how to use an emotion recognition engine to label the emotions of messages and then use that information to automatically generate replies."

[0742] These prompts allow the user to get guidance on the detailed operation and implementation of the system.

[0743] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0744] Program processing flow

[0745] Step 1:

[0746] The server collects unread messages from multiple communication services (e.g., email services, chat applications, social media) via a communication network. The input requires API authentication tokens and user IDs for each communication service. The server uses these to call each API and retrieve new and unread messages. The output includes metadata such as message content, sender, recipient, and timestamp, which are stored in a database.

[0747] Specific actions:

[0748] The server calls the Gmail API to retrieve the user's unread emails.

[0749] The server uses the chat application API to collect unread messages.

[0750] Step 2:

[0751] The server uses a natural language processing engine (e.g., SpaCy) to tokenize the messages it collects. The message text is required as input. The server divides the message text into words and outputs a sequence of tokenized words.

[0752] Specific actions:

[0753] The server uses SpaCy's nlp(text) function to tokenize the message text.

[0754] Step 3:

[0755] The server performs part-of-speech tagging on tokenized words. The input is a sequence of tokenized words. The server assigns a part-of-speech tag to each word, and the output is a sequence of words tagged with part-of-speech.

[0756] Specific actions:

[0757] The part-of-speech information of the tokens generated by SpaCy is extracted, and tags are assigned to each token.

[0758] Step 4:

[0759] The server performs semantic analysis of the text and generates a message summary. The input requires a sequence of words tagged with parts of speech. The server uses natural language processing techniques (e.g., NLTK) to extract key points and obtains a summary as output.

[0760] Specific actions:

[0761] The server uses the NLTK summarization algorithm to extract important sentences.

[0762] Step 5:

[0763] The server uses an emotion engine (e.g., Hugging Face's Transformers model) to recognize emotions in the collected messages. The message text is required as input. The emotion engine parses the text and outputs emotion labels such as positive, negative, or neutral.

[0764] Specific actions:

[0765] The server calls pipeline('sentiment-analysis') to retrieve sentiment labels from the message text.

[0766] Step 6:

[0767] The server uses a machine learning model (e.g., a classification model in Scikit-learn) to score the importance of collected messages. Input includes sentiment labels, important keywords, and sender information. The server calculates a score based on this information, and the output is the importance score.

[0768] Specific actions:

[0769] We will use a classification model from Scikit-learn to calculate an importance score for each message.

[0770] Step 7:

[0771] The server sorts messages based on their importance score. The input is a list of scored messages. The server sorts the messages by importance and outputs a sorted list of messages.

[0772] Specific actions:

[0773] The server sorts the message list based on its importance score, placing the most important messages at the top of the list.

[0774] Step 8:

[0775] The server automatically generates response content based on the user's past response data and recognized sentiment. Past response data and sentiment labels are required as input. A generative AI model (e.g., GPT-3) is used to generate a response based on the prompt, and the response content is obtained as output.

[0776] Specific actions:

[0777] This function calls the GPT-3 API and generates a response based on the prompt.

[0778] Input prompt example: "Generate an appropriate reply for the unread message."

[0779] Step 9:

[0780] The terminal displays the automatically generated reply received from the server to the user. The generated reply is required as input. The user reviews the reply on the terminal, makes any necessary corrections, and then receives the final confirmed reply as output.

[0781] Specific actions:

[0782] The user reviews and modifies the reply displayed on their device.

[0783] Step 10:

[0784] The user sends their confirmed reply to the server, which then uses the APIs of the respective communication services to actually send the final reply. The input requires the user's confirmed and revised final reply. The server sends the reply via the API and notifies the user when the transmission is complete.

[0785] Specific actions:

[0786] The server calls the Gmail API to send the email.

[0787] Send a message using the chat application API.

[0788] (Application Example 2)

[0789] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0790] In modern brick-and-mortar stores, customer inquiries are increasingly pouring in through various communication services. This means that store employees have to spend a considerable amount of time and effort reviewing and responding to messages appropriately. Furthermore, failure to respond immediately can lead to decreased customer satisfaction. Therefore, there is a need for technology that can efficiently manage messages and automatically generate appropriate responses.

[0791] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting messages from multiple communication services via a communication network, means for analyzing the collected messages using natural language processing technology and generating summaries, means for scoring the importance of the analyzed messages and sorting the messages based on importance, means for sending automatically generated reply content using the user's past reply data to the user terminal, means for the user to modify the automatically generated reply content displayed on the user terminal and send the final reply, means for applying emotion recognition technology to the collected messages and assigning emotion labels, and means for generating automatic reply content to respond to customer inquiry messages in a physical store, which is then modified and sent by the store staff. This makes it possible to respond to customer inquiries quickly and appropriately.

[0792] A "communication network" is an infrastructure for exchanging information between different locations, such as the internet or mobile phone networks.

[0793] A "message" is a unit of information that is sent and received in various formats, such as text, images, and videos.

[0794] "Natural language processing technology" is a technology that uses computers to analyze and understand human language.

[0795] A "means of generating summaries" is a technique for concisely summarizing the main content and key points of collected messages.

[0796] "Scoring importance" is the process of quantifying the importance of a message based on the content and sender information collected from that message.

[0797] "Emotion recognition technology" is a technology that detects emotions from data such as text and audio, and assigns specific emotion labels (positive, negative, neutral, etc.) to them.

[0798] "Automatically generated replies" are replies that a computer automatically creates based on the user's past reply patterns and templates.

[0799] A "user terminal" is a device (such as a smartphone or personal computer) that a user uses to receive or send information.

[0800] "A means of making revisions and sending a final reply" refers to the process by which a user reviews an automatically generated reply, makes changes as needed, and then sends the message.

[0801] A "physical store" is a store that exists physically and is a place where commercial transactions are conducted directly with customers.

[0802] An "inquiry message" is a message sent by a customer to a store or service provider with questions or requests.

[0803] This invention provides a system that enables users to efficiently manage messages from multiple communication services and provide prompt and appropriate responses. This invention significantly reduces the time and effort required to check and reply to messages. Furthermore, it enables appropriate responses that take into account the user's emotions.

[0804] System Configuration

[0805] This system includes the following main components:

[0806] 1. Collection method: The server collects messages from multiple communication services (e.g., email, chat applications, social media) via a communication network. The collected messages are stored in a database.

[0807] 2. Analysis Method: The server analyzes the collected messages using natural language processing (NLP) techniques and generates summaries. An NLP engine such as the TextBlob library is used for natural language processing.

[0808] 3. Scoring method: The server uses a machine learning model (e.g., scikit-learn's logistic regression model) to score the importance of messages. Based on the importance score, messages are sorted in order of importance.

[0809] 4. Emotion Recognition Method: The server uses emotion recognition technology (e.g., TextBlob library) to recognize the emotions of the collected messages and assign emotion labels such as positive, negative, or neutral.

[0810] 5. Automatic generation method: The server uses the user's past reply data and sends automatically generated reply content based on the recognized sentiment to the user's terminal. This response content is generated based on templates or corpora.

[0811] 6. Modification and Submission Method: The user makes modifications to the automatically generated reply displayed on their device and then submits the final reply. The modified reply is sent via the API of the respective communication service (e.g., email or chat application).

[0812] Hardware and software configuration

[0813] Hardware: Smartphones, PCs, servers

[0814] software:

[0815] Collection method: Requests library

[0816] Analysis method: TextBlob library

[0817] Scoring method: scikit-learn library

[0818] Emotion recognition method: TextBlob library

[0819] Automatic generation method: Template engine (e.g., Jinja2)

[0820] Modification / Sending Method: smtplib library (for sending emails)

[0821] Specific example

[0822] For example, in a physical store, if a customer sends a message inquiring about product availability via various communication services, this system will operate as follows:

[0823] 1. The server collects all inquiry messages using APIs from services such as Instagram, LINE, and Facebook Messenger.

[0824] 2. Apply an NLP engine to the collected messages to generate a summary.

[0825] 3. Use emotion recognition technology to recognize the emotions in a message and assign emotion labels.

[0826] 4. An importance score is generated, and important messages are displayed at the top of the list.

[0827] 5. The server automatically generates a reply based on the user's past reply data and sentiment recognition results, and sends it to the user's terminal.

[0828] 6. The user reviews the reply on their device, makes any necessary corrections, confirms it, and sends it.

[0829] Example of a prompt

[0830] "Please create an appropriate template for automatically sending replies inquiring about product availability."

[0831] With the above configuration, the system of the present invention is capable of responding quickly and appropriately to customer inquiries. This is expected to improve customer satisfaction and the work efficiency of store employees.

[0832] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0833] Step 1:

[0834] The server collects messages from multiple communication services via a communication network. The server uses APIs to retrieve unread messages from services such as email, chat applications, and social media. The input is the API authentication token for each service, and the output is the retrieved message and its metadata (sender, recipient, timestamp, content, etc.). The collected messages are stored in a database.

[0835] Step 2:

[0836] This process applies natural language processing (NLP) techniques to messages collected by the server to generate summaries. An NLP engine (e.g., TextBlob library) is used to tokenize and parse the messages. The input is the collected message body, and the output is the generated summary. The process extracts key parts from the message to create a concise summary.

[0837] Step 3:

[0838] The server uses sentiment recognition technology to recognize the sentiment of the collected messages. It uses the TextBlob library to extract sentiment labels (positive, negative, neutral, etc.) from the text within the messages. The input is the summarized message body, and the output is a summary with sentiment labels added. Appropriate sentiment labels are added according to the tone of the message.

[0839] Step 4:

[0840] The server uses a machine learning model to score the importance of collected messages. It uses a scikit-learn logistic regression model to calculate importance scores based on input data (summaries, sentiment labels, and metadata). The input consists of sentiment-labeled summaries and metadata, and the output is the importance score. Based on the scoring results, the messages are sorted in order of importance.

[0841] Step 5:

[0842] The server uses the user's past reply data to generate automatically generated replies based on the perceived sentiment. A template engine (e.g., Jinja2) is used to create the automated replies. Inputs include importance scores, sorted messages, and the user's past reply patterns; output is the automatically generated reply. Contextually appropriate replies are generated based on templates.

[0843] Step 6:

[0844] The user reviews the automatically generated reply displayed on their device, makes any necessary corrections, and sends the final reply. The user checks the reply content using a smartphone or computer and makes corrections as needed. The input is the automatically generated reply, and the output is the corrected, final reply. After the user confirms and approves the reply, it is sent via the API of the respective communication service.

[0845] Through the processing steps described above, the server can efficiently execute a series of processes from message collection to automatic reply generation. For example, in response to a message inquiring about product inventory status, an automatic reply can be quickly generated, allowing store staff to modify and send it as needed.

[0846] Example prompt: "Please create an appropriate template for automatically sending a message inquiring about product availability."

[0847] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0848] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0849] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0850] [Third Embodiment]

[0851] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0852] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0853] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0854] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0855] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0856] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0857] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0858] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0859] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0860] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0861] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0862] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0863] This invention provides a system that automatically collects messages from multiple communication services used by a user, summarizes and evaluates their importance using natural language processing technology, and then provides the user with automatically generated replies. This system reduces the time and effort users spend checking and replying to messages.

[0864] System Configuration

[0865] This system includes the following main components:

[0866] 1. Collection method: The server collects messages from multiple communication services (e.g., email services, chat applications, social media) via a communication network.

[0867] 2. Analysis method: The server analyzes the collected messages using natural language processing technology and generates a summary.

[0868] 3. Scoring method: The server scores the importance of the analyzed messages and sorts them based on importance.

[0869] 4. Automatic generation method: The server uses the user's past reply data to automatically generate a reply and send it to the user's terminal.

[0870] 5. Correction and Submission Method: The user reviews the automatically generated reply displayed on their device, makes corrections as needed, and then submits the final reply.

[0871] Program processing

[0872] Collection methods

[0873] The server retrieves unread messages via APIs from Gmail, chat applications, and other services through the communication network. The server uses authentication tokens from each service to access user accounts and collect new and unread messages. The collected messages are stored in a database, which also stores metadata such as sender, recipient, and timestamp, in addition to the message content.

[0874] Analysis means

[0875] The server applies a natural language processing engine to the collected messages. This engine tokenizes the messages, tags them by part of speech, and analyzes the meaning of the sentences. As a result, a summary is generated and important information is extracted.

[0876] Scoring methods

[0877] The server uses a machine learning model to score the importance of the collected messages. This scoring takes into account factors such as the frequency of important keywords, the sender's role and relationship, and the urgency of past interactions. Based on the importance score, messages are sorted in order of importance.

[0878] automatic generation means

[0879] The server learns from the user's past reply data and creates automatically generated replies suitable for similar situations. These replies are sent to the user's device in JSON format, and the user can review them as drafts.

[0880] Correction / Transmission Methods

[0881] The terminal displays the automatically generated reply received from the server to the user. The user reviews the reply using the terminal's interface, makes any necessary corrections, and then confirms the final reply. The confirmed reply is sent to the server and then actually transmitted through the respective communication services.

[0882] Specific example

[0883] For example, if a user receives many unread emails or chat messages in a day, this system will work as follows:

[0884] 1. Collection method: The server uses the Gmail API and the chat app API to collect all unread messages.

[0885] 2. Analysis method: An NLP engine is applied to the collected messages to generate a summary.

[0886] 3. Scoring method: An importance score is calculated, and important messages are displayed at the top of the list.

[0887] 4. Automatic generation method: The server automatically generates a reply based on the user's past reply data and sends it to the user's terminal.

[0888] 5. Correction and Submission Method: The user reviews the reply on their device, makes any necessary corrections, confirms the message, and sends it.

[0889] This system significantly improves user work efficiency and reduces the time required to check and reply to messages.

[0890] The following describes the processing flow.

[0891] Step 1:

[0892] The server collects messages from communication services via the communication network. Specifically, it calls APIs of Gmail and chat applications to retrieve new and unread messages. In doing so, it uses API authentication tokens from each service to access the user's account.

[0893] Step 2:

[0894] The server stores the collected messages in a database. When saving, not only the message content but also metadata such as the sender, recipient, and timestamp are saved together.

[0895] Step 3:

[0896] The server initiates natural language processing on the stored messages. First, it tokenizes the messages, splitting them into individual words. Next, it tags each word with its part of speech and analyzes the meaning of the entire sentence. Based on this analysis, it generates a summary of the message.

[0897] Step 4:

[0898] The server uses a machine learning model to score importance based on the generated summary and message metadata. The scoring takes into account factors such as the frequency of important keywords, the relationship between the sender and receiver, and the urgency determined from past interactions.

[0899] Step 5:

[0900] The server sorts the scored messages by importance. Messages with higher scores are placed higher in the list, creating a list where important messages are displayed preferentially.

[0901] Step 6:

[0902] The server uses a model that has learned from the user's past reply data to create automatically generated replies. These replies are generated based on patterns in the user's past interactions and common reply templates.

[0903] Step 7:

[0904] The server sends the generated reply in JSON format to the user's device. The device receives this reply and displays it to the user.

[0905] Step 8:

[0906] The user reviews the automatically generated reply displayed on their device. They make any necessary corrections and finalize the reply.

[0907] Step 9:

[0908] The user submits their finalized reply by clicking the send button. The server then sends this reply through the API of the respective communication service (e.g., Gmail or chat application).

[0909] Through these steps, the system significantly reduces the time users spend checking and replying to messages, enabling more efficient communication.

[0910] (Example 1)

[0911] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0912] Currently, many users receive a large volume of messages daily through multiple communication services, resulting in the problem of significant time and effort being required to review and respond to them. Furthermore, the risk of overlooking important messages increases, highlighting the need for efficient message management. To address this challenge, a system is needed that can effectively automate message collection, summarization, importance assessment, and automatic responses.

[0913] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0914] In this invention, the server includes means for collecting messages from multiple communication services via a communication network; means for analyzing the collected messages using natural language processing technology and generating summaries; means for scoring the importance of the analyzed messages and sorting the messages based on importance; means for sending automatically generated reply content using the user's past reply data to the user terminal; means for the user to modify the automatically generated reply content displayed on the user terminal and send the final reply; means for storing metadata such as the sender, recipient, timestamp, and content of the collected messages; means for using a machine learning model to generate importance scores; means for tokenizing, tagging parts of speech, and analyzing sentences on the collected messages to extract important information; means for sending the automatically generated reply content in JSON format to the user terminal; and means for saving the reply content after user input and actually sending it via the communication service. As a result, users can significantly reduce the time required to check and reply to messages and efficiently manage important messages.

[0915] A "communication network" is the infrastructure used to send and receive data, and includes the internet and internal networks.

[0916] "Multiple communication services" refers to messaging services provided by different providers or applications, including, for example, email, chat apps, and social media.

[0917] A "message" is a unit of information that takes the form of text, images, audio, video, etc., and is sent from a sender to a recipient via a communication service.

[0918] "Natural language processing technology" refers to technologies for understanding, interpreting, and generating human language using computers, and includes processes such as tokenization, part-of-speech tagging, and semantic analysis.

[0919] A "summary" is information that extracts the most important information from the original message and presents it in a concise format.

[0920] "Importance" is a numerical value or rank that evaluates how important the content of a message is to the user.

[0921] "Scoring" is the process of evaluating the importance of a message and quantifying or ranking it.

[0922] "User's past reply data" refers to the history of reply messages previously sent by the user, and serves as reference data for creating automatically generated new reply content.

[0923] "Automatically generated reply content" refers to the content of a reply message that the system automatically creates based on the user's past reply data and other criteria.

[0924] A "user terminal" refers to a device such as a computer or smartphone used by a user, which can be used to check and reply to messages.

[0925] "Metadata" refers to information about the message itself, including attributes such as the sender, recipient, timestamp, and content.

[0926] A "machine learning model" is an algorithm or statistical model that uses large amounts of data to learn patterns and perform predictions or classifications.

[0927] "Tokenization" is the process of dividing text into meaningful units (tokens).

[0928] "Part-of-speech tagging" is the process of assigning a part of speech (noun, verb, adjective, etc.) to each token.

[0929] "Sentence analysis" is the process of analyzing the grammatical and semantic structure of a text.

[0930] JSON format is a lightweight data exchange format that represents data as key-value pairs.

[0931] This invention provides a system for efficiently managing messages by automatically collecting messages from multiple communication services used daily by users, and analyzing and organizing them using natural language processing technology. This system consists of a server and a user terminal and includes the following means.

[0932] System hardware and software

[0933] server

[0934] The server has the following functions:

[0935] A function that collects messages via a communication network. Specific examples include using the Gmail API or chat application APIs.

[0936] A function to store collected messages in a database. This uses a database management system such as MySQL.

[0937] A function that applies natural language processing technology. Google's BERT can be used.

[0938] A function that generates a summary based on the analysis results.

[0939] A feature that uses machine learning models to score importance and sort messages accordingly.

[0940] This feature automatically generates replies based on the user's past response data. It utilizes machine learning libraries such as TensorFlow and PyTorch for this purpose.

[0941] A function that sends automatically generated reply content to the user's terminal in JSON format.

[0942] User terminal

[0943] The user terminal has the following functions:

[0944] A function that displays automatically generated replies received from the server to the user.

[0945] A web application is used as an interface for users to review replies and make corrections as needed.

[0946] A function that sends the revised reply to the server and then finally sends the message through each communication service.

[0947] Data processing and data calculation

[0948] The server collects messages using the APIs of various communication services and stores them in a database. Metadata such as sender, recipient, timestamp, and content is also stored. Next, a natural language processing engine (such as BERT) is used to analyze the messages, extract important information, and generate summaries. Furthermore, a machine learning model is used to score the importance of the messages and sort them accordingly.

[0949] Based on the user's past reply data, an automatically generated reply is created and sent to the user's terminal in JSON format. The user reviews the content displayed on their terminal, makes any necessary corrections, and sends the final message to the server. The server then sends this message to the respective communication services, and the message is actually sent.

[0950] Specific example

[0951] For example, if a user receives many unread emails in their Gmail account, the server uses the Gmail API to collect these messages and store them in a database. A natural language processing engine is applied to the collected messages to generate summaries. Next, a machine learning model is used to calculate importance scores and sort the messages so that the most important ones are displayed first. The automatically generated reply is sent to the user's terminal in JSON format, and after the user reviews and modifies it, the final reply is sent to the communication service.

[0952] Example of a prompt

[0953] Examples of prompt statements to input into a generative AI model include:

[0954] "Collect unread messages received by users, generate summaries, assess their importance, and create prompts to generate automated replies."

[0955] The above describes the "modes for carrying out the invention" of the system related to the present invention. This system allows users to significantly reduce the time and effort required to check and reply to messages.

[0956] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0957] Step 1: Collecting Messages

[0958] Server operation:

[0959] Input: Use user authentication information (such as an OAuth token) to send requests to the APIs of various communication services (Gmail, chat apps, social media, etc.).

[0960] Data processing: Receive unread message data obtained from communication services and convert it to the required format.

[0961] Output: Content and metadata of unread messages (sender, recipient, timestamp, etc.).

[0962] Specific operation: The server sends a request to the Gmail API, retrieves unread messages, and stores them in the "messages" table in the database.

[0963] Step 2: Message Analysis

[0964] Server operation:

[0965] Input: Content and metadata of unread messages stored in the database.

[0966] Data processing: Apply a natural language processing engine (such as BERT) to tokenize messages, tag them by part of speech, and perform semantic analysis of the sentences.

[0967] Output: Message summary and extracted key information.

[0968] Specific operation: The server analyzes the message "Today's meeting is at 3 PM" and generates a summary such as "Meeting change."

[0969] Step 3: Importance scoring

[0970] Server operation:

[0971] Input: Summary and metadata of the analyzed message (sender's job title and relationship, urgency level, etc.).

[0972] Data processing: An importance score is calculated using a machine learning model. Scoring factors include keyword frequency and the urgency of past interactions.

[0973] Output: Importance score and a list of messages sorted by importance.

[0974] Specific operation: The server evaluates emails from superiors regarding "meeting changes" with a high score and places them higher in the list.

[0975] Step 4: Automatically generate reply content

[0976] Server operation:

[0977] Input: User's past reply data and a list of messages sorted by importance.

[0978] Data processing: Generate appropriate response content based on past response data. This is done using machine learning libraries (e.g., TensorFlow or PyTorch).

[0979] Output: Auto-generated reply content in JSON format.

[0980] Specific operation: The server generates a new "Understood" message based on the user's previous "Understood" reply and sends it to the user's terminal in JSON format.

[0981] Step 5: Presenting the response to the user

[0982] Device operation:

[0983] Input: Automated response content in JSON format received from the server.

[0984] Data processing: Convert the received JSON data into a user-friendly format and display it on the interface.

[0985] Output: The reply displayed to the user.

[0986] Specific action: The device displays an automatically generated reply message to the user saying "Understood."

[0987] Step 6: User modification and submission

[0988] User actions:

[0989] Input: The automatically generated reply content that was displayed.

[0990] Data processing: Modify the reply content as needed.

[0991] Output: The revised and final reply.

[0992] Specific action: The user changes "Understood" to "Understood. I have also confirmed the meeting time," and then presses the send button.

[0993] Step 7: Send the revised reply.

[0994] Server operation:

[0995] Input: The corrected reply received from the user's terminal.

[0996] Data processing: The modified response content is sent via the API of each communication service.

[0997] Output: The final reply message sent to each communication service.

[0998] Specific action: The server actually sends the corrected reply, "Understood. I've also confirmed the meeting time," via the Gmail API.

[0999] (Application Example 1)

[1000] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1001] In modern logistics centers, numerous employees exchange information using a variety of communication methods, resulting in a massive volume of messages, which poses a significant challenge. In this environment, quickly identifying and responding appropriately to important messages is crucial, but doing so manually is time-consuming and labor-intensive, hindering efficient operations. Furthermore, while automated generation of replies based on relevant past response data is desirable, effective methods for achieving this have lacked.

[1002] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1003] In this invention, the server includes means for collecting messages from multiple communication services via a communication network; means for analyzing the collected messages using natural language processing technology and generating summaries; means for scoring the importance of the analyzed messages and sorting the messages based on importance; means for sending automatically generated replies to a user terminal using the user's past reply data; means for the user to modify the automatically generated replies displayed on the user terminal and send the final reply; and means for automatically collecting and analyzing messages from multiple communication tools used by employees in the logistics center and automatically generating appropriate replies. This makes it possible to efficiently manage a vast number of messages in the logistics center, respond quickly to important messages, and improve work efficiency by automatically generating appropriate replies.

[1004] A "communication network" is the infrastructure used to connect computers to each other and to send and receive data.

[1005] "Communication services" refer to platforms for sending and receiving digital messages, such as email, chat applications, and social media.

[1006] "Message" refers to a unit of text, images, audio, and other data transmitted via electronic means of communication.

[1007] Natural language processing is a branch of computer science that analyzes and understands human language, and is often used for tasks such as text summarization and sentiment analysis.

[1008] A "summary" is a format in which the content of the original message is shortened and only the important information is extracted.

[1009] "Scoring" refers to the process of numerically evaluating the importance of a message, which makes it possible to prioritize messages.

[1010] A "user terminal" is a device used by a user to operate something, and includes smartphones, tablets, and personal computers.

[1011] "Reply data" refers to a history of messages previously sent by the user.

[1012] "Automatic generation" refers to the process of creating messages and replies based on pre-programmed algorithms, without human intervention.

[1013] A "logistics center" is a facility for storing, sorting, and shipping goods, and its purpose is to enable efficient logistics management.

[1014] "Employees" refers to workers who work at the logistics center and are responsible for tasks such as information gathering, sorting, shipping, and management.

[1015] "Communication tools" refer to applications and software used to send and receive messages, such as email, chat applications, and notifications from IoT devices.

[1016] This invention is a system for achieving efficient message management and rapid response within a logistics center. The system configuration and specific embodiments are described below.

[1017] System Configuration

[1018] The system includes the following main components:

[1019] 1. Server: Collects messages from multiple communication services via the communication network, and analyzes, summarizes, scores, and filters them using natural language processing technology. It also creates automatically generated replies based on the user's past reply data and sends them to the user's terminal.

[1020] 2. User terminal: Used by employees at the logistics center to view summaries of messages sent from the server, automatically generated replies, make corrections, and send final replies.

[1021] 3. Communication Network: This is the infrastructure that connects servers and user terminals and allows the use of multiple communication service APIs.

[1022] Hardware and software configuration

[1023] hardware

[1024] User devices: Smartphones, tablets, personal computers, etc.

[1025] Server: Responsible for data processing as the central management system.

[1026] software

[1027] Python: Used for implementing natural language processing and machine learning models.

[1028] Generative AI models: GPT-3, BERT, etc., used for text generation and information extraction.

[1029] APIs from various communication services, such as the Gmail API, Slack API, and IoT device APIs, are used to collect messages.

[1030] Database: Used for managing messages and metadata.

[1031] Processing flow

[1032] The server first collects new and unread messages through the APIs of each communication service. Next, it applies a natural language processing engine (e.g., SpaCy) to the collected messages to generate summaries. Then, it uses a machine learning model (e.g., importance scoring using TF-IDF) to score the importance of the messages and sorts them in order of importance.

[1033] Subsequently, the server automatically generates appropriate replies using a generative AI model (e.g., GPT-3) based on the user's past reply data. This generated reply is sent to the user's terminal in JSON format, allowing the user to review, revise, and submit their final reply as a draft.

[1034] Specific example

[1035] For example, if an employee at a logistics center receives a large number of unread messages and notifications, this system will work as follows:

[1036] 1. Message Collection and Analysis: The server uses the Gmail API, Slack API, and IoT device APIs to collect all unread messages and notifications, and generates summaries using a natural language processing engine.

[1037] 2. Scoring and Sorting: Use a machine learning model to calculate importance scores and prioritize and sort important messages.

[1038] 3. Automatic generation of replies: The server uses an AI model to automatically generate appropriate replies based on the collected messages.

[1039] 4. Review and correction on the user's device: The user reviews the reply on their device, makes any necessary corrections, and then confirms and sends the final reply.

[1040] Examples of prompts for generative AI models

[1041] example:

[1042] Please generate a response appropriate to the following situation:

[1043] situation:

[1044] We received an inquiry from a customer stating that their ordered item has not yet arrived. Upon checking the tracking information, we found that the delivery is delayed.

[1045] reply:

[1046] This system streamlines message management within the logistics center, significantly improving employee work efficiency. Furthermore, it enables quick responses to important messages, contributing to increased customer satisfaction.

[1047] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1048] Step 1:

[1049] The server collects messages from multiple communication services (such as Gmail, Slack, and IoT devices) via a communication network. In this step, it uses APIs to retrieve new and unread messages and saves them to a local database. The input is the message data obtained from each communication service, and the output is the message data stored in the local database.

[1050] Step 2:

[1051] The server applies natural language processing techniques to the collected messages to generate summaries. In this step, libraries such as SpaCy are used to tokenize the messages, extract important information, and generate summaries. The input is message data obtained from a local database, and the output is the generated summary data. Specifically, the first few sentences of the message body are extracted and used as the summary.

[1052] Step 3:

[1053] The server scores and sorts the importance of the analyzed messages. In this step, a machine learning model (e.g., TF-IDF scoring) is used to evaluate the importance of the messages. The input is summarized message data, and the output is a list of messages that have been scored and sorted by importance. Specifically, the TF-IDF score for each message is calculated, and the messages are sorted based on that score.

[1054] Step 4:

[1055] The server uses the user's past reply data to create automatically generated replies using a generative AI model (e.g., GPT-3). In this step, the reply is generated based on the prompt. The input is scored message data and the user's past reply data, and the output is the generated reply. Specifically, the prompt is input to the generative AI model, and the generated reply is retrieved.

[1056] Step 5:

[1057] The server automatically generates a reply and sends it to the user's terminal. In this step, the reply generated in JSON format is transferred to the user's terminal. The input is the generated reply, and the output is a notification that the transmission to the user's terminal is complete. Specifically, the reply is sent using the user's terminal's identification information.

[1058] Step 6:

[1059] The user terminal displays the automatically generated reply to the user. In this step, the reply is displayed on the interface, and the user is provided with the ability to make modifications. The input is the automatically generated reply sent from the server, and the output is the reply that has been reviewed and modified by the user. Specifically, the reply is rendered on the display interface.

[1060] Step 7:

[1061] The user makes final revisions to their reply and sends it. In this step, the user reviews and confirms the revised reply, and then sends it via the server through the respective communication services. The input is the reply revised by the user, and the output is the final reply message sent. Specifically, based on the user's confirmation action, the message is sent through the respective communication service APIs.

[1062] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1063] This invention is a system that automatically collects messages from multiple communication services used by a user, analyzes them using natural language processing technology and an emotion engine to summarize and evaluate their importance, and provides the user with automatically generated replies. This system reduces the time and effort users spend checking and replying to messages, and also enables appropriate responses that take into account the user's emotions.

[1064] System Configuration

[1065] This system includes the following main components:

[1066] 1. Collection method: The server collects messages from multiple communication services (email services, chat applications, social media) via a communication network.

[1067] 2. Analysis method: The server analyzes the collected messages using natural language processing technology and generates a summary.

[1068] 3. Scoring method: The server scores the importance of the analyzed messages and sorts them based on importance.

[1069] 4. Emotion Recognition Method: The server uses an emotion engine to recognize emotions from collected messages.

[1070] 5. Automatic generation means: The server automatically generates a reply based on the user's past reply data and recognized emotions and sends it to the user's terminal.

[1071] 6. Correction and Submission Method: The user reviews the automatically generated reply displayed on their device, makes corrections as needed, and then submits the final reply.

[1072] Program processing

[1073] Collection methods

[1074] The server retrieves unread messages via APIs from services such as Gmail and chat applications through the communication network. It accesses user accounts using API authentication tokens for each service to collect new and unread messages. The collected messages are stored in a database, which stores not only the message content but also metadata such as sender, recipient, and timestamp.

[1075] Analysis means

[1076] The server applies a natural language processing engine to the collected messages. First, the messages are tokenized and split into individual words. Next, the part of speech of each word is identified, the meaning of the sentence is analyzed, and a message summary is generated.

[1077] emotion recognition means

[1078] The server uses an emotion engine to recognize the sentiment of a message. The emotion engine extracts emotions from the text within the message and assigns sentiment labels such as positive, negative, and neutral. The recognized sentiment influences importance scoring and automated reply generation.

[1079] Scoring methods

[1080] The server uses a machine learning model to score the importance of the collected messages. The scoring considers factors such as the frequency of important keywords, the sender's role and relationship, and the results of sentiment recognition. Based on the importance score, messages are sorted in order of importance.

[1081] automatic generation means

[1082] The server automatically generates a response based on the user's past reply data and perceived emotions. This response is generated based on patterns of past interactions and common reply templates. In particular, it generates a response with an appropriate tone and content that takes the results of emotion recognition into account.

[1083] Correction / Transmission Methods

[1084] The device displays the automatically generated reply received from the server to the user. The user reviews the reply on the device, makes any necessary corrections, and then confirms the final reply. The confirmed reply is sent to the server and then actually transmitted through the APIs of each communication service (e.g., Gmail or chat applications).

[1085] Specific example

[1086] For example, if a user receives many unread emails or chat messages in a day, this system will work as follows:

[1087] 1. Collection method: The server uses the Gmail API and the chat app API to collect all unread messages.

[1088] 2. Analysis method: An NLP engine is applied to the collected messages to generate a summary.

[1089] 3. Emotion Recognition Method: The emotion engine recognizes the emotion of the message and assigns an emotion label.

[1090] 4. Scoring method: An importance score is calculated, and important messages are displayed at the top of the list.

[1091] 5. Automatic generation method: The server automatically generates a reply based on the user's past reply data and sentiment recognition results, and sends it to the user's terminal.

[1092] 6. Correction and Submission Method: The user reviews the reply on their device, makes any necessary corrections, confirms the message, and sends it.

[1093] This system significantly improves user productivity, reducing the time required to check and reply to messages, and enabling appropriate responses that take user emotions into consideration.

[1094] The following describes the processing flow.

[1095] Step 1:

[1096] The server collects messages from communication services via the communication network. Specifically, it calls APIs of Gmail and chat applications to retrieve new and unread messages. In doing so, it uses API authentication tokens from each service to access the user's account.

[1097] Step 2:

[1098] The server stores the collected messages in a database. When saving, not only the message content but also metadata such as the sender, recipient, and timestamp are saved together.

[1099] Step 3:

[1100] The server initiates natural language processing (NLP) on the stored messages. First, the messages are tokenized and split into individual words. Next, part-of-speech tags are added to each word, and the meaning of the entire sentence is analyzed to generate a summary of the message.

[1101] Step 4:

[1102] The server uses an emotion engine to recognize the emotion of a message. The emotion engine extracts emotion labels such as positive, negative, and neutral from the message and adds this emotion information to the message data.

[1103] Step 5:

[1104] The server uses a machine learning model to score the importance of the collected messages. The scoring considers factors such as the frequency of important keywords, the sender's role and relationship, and the results of sentiment recognition. Based on the importance score, messages are sorted in order of importance.

[1105] Step 6:

[1106] The server automatically generates replies based on the user's past reply data and sentiment recognition results. The generated replies are based on patterns of past interactions and common reply templates, and include replies with an appropriate tone and content, taking into account the sentiment recognition results in particular.

[1107] Step 7:

[1108] The server sends the generated reply in JSON format to the user's device. The device receives this reply and displays it to the user.

[1109] Step 8:

[1110] The user reviews the automatically generated reply displayed on their device. They make any necessary corrections and finalize the reply.

[1111] Step 9:

[1112] The user submits their finalized reply by clicking the send button. The server then sends this reply through the API of the respective communication service (e.g., Gmail or chat application).

[1113] Through this series of steps, users can significantly reduce the time spent checking and responding to messages, and also be able to provide appropriate responses that take into account the user's emotions.

[1114] (Example 2)

[1115] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1116] In modern society, users utilize multiple communication services (email, chat applications, social media, etc.) and receive a large volume of messages daily. However, responding quickly and appropriately to all messages is difficult, and managing and replying to messages consumes a significant amount of time and effort. Furthermore, accurately understanding the content, importance, and sentiment of a message, and responding accordingly, requires high skill. Therefore, there is a need for technology that can quickly generate appropriate responses and reduce the user's time and effort.

[1117] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting messages from multiple communication services via a communication network, means for storing metadata (sender, recipient, timestamp, content) of the collected messages, means for tokenizing the collected messages using natural language processing technology, assigning part-of-speech tags, and generating summaries, means for recognizing the sentiment of messages using a sentiment engine and labeling the results, means for scoring the importance of messages analyzed using a machine learning model and sorting messages based on importance, means for sending automatically generated reply content based on the user's past reply data and recognized sentiment to the user terminal, and means for the user to modify the automatically generated reply content displayed on the user terminal and send the final reply. This enables the user to respond quickly and appropriately to a large volume of messages, as well as significantly reduce the user's time and effort.

[1118] A "communication network" refers to a computer network used for sending and receiving data.

[1119] "Communication services" refer to services that allow users to exchange information, such as email, chat applications, and social media.

[1120] A "message" refers to information such as text, images, and files sent by a user.

[1121] "Metadata" refers to supplementary information accompanying a message, including attribute data such as sender, recipient, timestamp, and content.

[1122] "Natural language processing technology" refers to the technology used to process natural language used by humans using computers.

[1123] "Tokenization" refers to the process of dividing a continuous text into words or phrases.

[1124] "Part-of-speech tagging" refers to the process of identifying which part of speech each tokenized word belongs to, such as a noun, verb, or adjective.

[1125] A "summary" refers to a text that extracts the main points of the original message and presents them concisely.

[1126] An "emotion engine" refers to an algorithm that recognizes and labels emotions from text data.

[1127] A "machine learning model" refers to an algorithm that learns patterns from data and uses those learning results to analyze and predict new data.

[1128] "Importance scoring" refers to the process of numerically evaluating the importance of a message.

[1129] "Sorting" refers to the process of ordering messages based on their importance score.

[1130] "Automatically generated replies" refer to reply messages created by a computer based on the user's past reply patterns and perceived emotions.

[1131] "User terminal" refers to a device (such as a smartphone, tablet, or personal computer) that a user uses to access the system.

[1132] "Correction" refers to the act of a user making changes or additions to automatically generated replies.

[1133] "Final reply" refers to the reply message that the user has finalized after making revisions.

[1134] This invention is a system that automatically collects messages from multiple communication services used by a user, analyzes them using natural language processing technology and an emotion engine to perform summarization and importance evaluation, and provides the user with automatically generated replies.

[1135] This system includes the following main components:

[1136] 1. Collection Method: The server collects messages from multiple communication services (email services, chat applications, social media) via the communication network. Specifically, it uses the Gmail API and chat application APIs to retrieve users' unread messages. It accesses user accounts using API authentication tokens for each service to collect new and unread messages. The collected messages are stored in a database, and in addition to the message content, metadata such as sender, recipient, and timestamp is also stored.

[1137] 2. Analysis Method: The server tokenizes the collected messages using natural language processing techniques (e.g., SpaCy, NLTK) and tags them by part of speech. Tokenization divides the message text into words, and part-of-speech tagging identifies the part of speech of each word. Next, the server performs semantic analysis of the sentences and generates a summary of the messages. The NLTK summarization algorithm is used for this purpose.

[1138] 3. Emotion Recognition Method: The server uses an emotion engine (e.g., Hugging Face's Transformers model) to recognize the emotion of a message. The emotion engine extracts emotions from the text within the message and assigns emotion labels such as positive, negative, and neutral. The recognized emotion influences importance scoring and automatic reply generation.

[1139] 4. Scoring Method: The server uses a machine learning model (e.g., a classification model from Scikit-learn) to score the importance of the collected messages. The scoring considers factors such as the frequency of important keywords, the sender's role and relationship, and the results of sentiment recognition. Based on the importance score, messages are sorted in order of importance.

[1140] 5. Automatic generation method: The server automatically generates response content based on the user's past response data and recognized emotions. This response content is generated based on patterns of past interactions and general response templates. In particular, responses with appropriate tone and content are generated, taking into account the results of emotion recognition. The server generates responses using a generation AI model (e.g., GPT-3).

[1141] 6. Correction and Sending Method: The terminal displays the automatically generated reply received from the server to the user. The user reviews the reply on the terminal, makes any necessary corrections, and then confirms the final reply. The confirmed reply is sent to the server and actually sent through the APIs of each communication service (e.g., Gmail or chat applications).

[1142] As a concrete example, if a user receives many unread emails and chat messages in a day, this system will work as follows:

[1143] 1. The server uses the Gmail API and the chat app API to collect all unread messages.

[1144] 2. A natural language processing engine is applied to the collected messages to generate a summary.

[1145] 3. The emotion engine recognizes the emotion of the message and assigns an emotion label.

[1146] 4. An importance score is calculated, and important messages are displayed at the top of the list.

[1147] 5. The server automatically generates a reply based on the user's past reply data and sentiment recognition results, and sends it to the user's terminal.

[1148] 6. The user reviews the reply on their device, makes any necessary corrections, confirms it, and sends it.

[1149] Examples of input prompts for a generative AI model are shown below:

[1150] 1. "Please describe the process for automatically collecting unread messages from email and chat applications, generating summaries, sorting them based on importance scores, and automatically generating appropriate replies."

[1151] 2. "Explain how to use an emotion recognition engine to label the emotions of messages and then use that information to automatically generate replies."

[1152] These prompts allow the user to get guidance on the detailed operation and implementation of the system.

[1153] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1154] Program processing flow

[1155] Step 1:

[1156] The server collects unread messages from multiple communication services (e.g., email services, chat applications, social media) via a communication network. The input requires API authentication tokens and user IDs for each communication service. The server uses these to call each API and retrieve new and unread messages. The output includes metadata such as message content, sender, recipient, and timestamp, which are stored in a database.

[1157] Specific actions:

[1158] The server calls the Gmail API to retrieve the user's unread emails.

[1159] The server uses the chat application API to collect unread messages.

[1160] Step 2:

[1161] The server uses a natural language processing engine (e.g., SpaCy) to tokenize the messages it collects. The message text is required as input. The server divides the message text into words and outputs a sequence of tokenized words.

[1162] Specific actions:

[1163] The server uses SpaCy's nlp(text) function to tokenize the message text.

[1164] Step 3:

[1165] The server performs part-of-speech tagging on tokenized words. The input is a sequence of tokenized words. The server assigns a part-of-speech tag to each word, and the output is a sequence of words tagged with part-of-speech.

[1166] Specific actions:

[1167] The part-of-speech information of the tokens generated by SpaCy is extracted, and tags are assigned to each token.

[1168] Step 4:

[1169] The server performs semantic analysis of the text and generates a message summary. The input requires a sequence of words tagged with parts of speech. The server uses natural language processing techniques (e.g., NLTK) to extract key points and obtains a summary as output.

[1170] Specific actions:

[1171] The server uses the NLTK summarization algorithm to extract important sentences.

[1172] Step 5:

[1173] The server uses an emotion engine (e.g., Hugging Face's Transformers model) to recognize emotions in the collected messages. The message text is required as input. The emotion engine parses the text and outputs emotion labels such as positive, negative, or neutral.

[1174] Specific actions:

[1175] The server calls pipeline('sentiment-analysis') to retrieve sentiment labels from the message text.

[1176] Step 6:

[1177] The server uses a machine learning model (e.g., a classification model in Scikit-learn) to score the importance of collected messages. Input includes sentiment labels, important keywords, and sender information. The server calculates a score based on this information, and the output is the importance score.

[1178] Specific actions:

[1179] We will use a classification model from Scikit-learn to calculate an importance score for each message.

[1180] Step 7:

[1181] The server sorts messages based on their importance score. The input is a list of scored messages. The server sorts the messages by importance and outputs a sorted list of messages.

[1182] Specific actions:

[1183] The server sorts the message list based on its importance score, placing the most important messages at the top of the list.

[1184] Step 8:

[1185] The server automatically generates response content based on the user's past response data and recognized sentiment. Past response data and sentiment labels are required as input. A generative AI model (e.g., GPT-3) is used to generate a response based on the prompt, and the response content is obtained as output.

[1186] Specific actions:

[1187] This function calls the GPT-3 API and generates a response based on the prompt.

[1188] Input prompt example: "Generate an appropriate reply for the unread message."

[1189] Step 9:

[1190] The terminal displays the automatically generated reply received from the server to the user. The generated reply is required as input. The user reviews the reply on the terminal, makes any necessary corrections, and then receives the final confirmed reply as output.

[1191] Specific actions:

[1192] The user reviews and modifies the reply displayed on their device.

[1193] Step 10:

[1194] The user sends their confirmed reply to the server, which then uses the APIs of the respective communication services to actually send the final reply. The input requires the user's confirmed and revised final reply. The server sends the reply via the API and notifies the user when the transmission is complete.

[1195] Specific actions:

[1196] The server calls the Gmail API to send the email.

[1197] Send a message using the chat application API.

[1198] (Application Example 2)

[1199] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1200] In modern brick-and-mortar stores, customer inquiries are increasingly pouring in through various communication services. This means that store employees have to spend a considerable amount of time and effort reviewing and responding to messages appropriately. Furthermore, failure to respond immediately can lead to decreased customer satisfaction. Therefore, there is a need for technology that can efficiently manage messages and automatically generate appropriate responses.

[1201] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting messages from multiple communication services via a communication network, means for analyzing the collected messages using natural language processing technology and generating summaries, means for scoring the importance of the analyzed messages and sorting the messages based on importance, means for sending automatically generated reply content using the user's past reply data to the user terminal, means for the user to modify the automatically generated reply content displayed on the user terminal and send the final reply, means for applying emotion recognition technology to the collected messages and assigning emotion labels, and means for generating automatic reply content to respond to customer inquiry messages in a physical store, which is then modified and sent by the store staff. This makes it possible to respond to customer inquiries quickly and appropriately.

[1202] A "communication network" is an infrastructure for exchanging information between different locations, such as the internet or mobile phone networks.

[1203] A "message" is a unit of information that is sent and received in various formats, such as text, images, and videos.

[1204] "Natural language processing technology" is a technology that uses computers to analyze and understand human language.

[1205] A "means of generating summaries" is a technique for concisely summarizing the main content and key points of collected messages.

[1206] "Scoring importance" is the process of quantifying the importance of a message based on the content and sender information collected from that message.

[1207] "Emotion recognition technology" is a technology that detects emotions from data such as text and audio, and assigns specific emotion labels (positive, negative, neutral, etc.) to them.

[1208] "Automatically generated replies" are replies that a computer automatically creates based on the user's past reply patterns and templates.

[1209] A "user terminal" is a device (such as a smartphone or personal computer) that a user uses to receive or send information.

[1210] "A means of making revisions and sending a final reply" refers to the process by which a user reviews an automatically generated reply, makes changes as needed, and then sends the message.

[1211] A "physical store" is a store that exists physically and is a place where commercial transactions are conducted directly with customers.

[1212] An "inquiry message" is a message sent by a customer to a store or service provider with questions or requests.

[1213] This invention provides a system that enables users to efficiently manage messages from multiple communication services and provide prompt and appropriate responses. This invention significantly reduces the time and effort required to check and reply to messages. Furthermore, it enables appropriate responses that take into account the user's emotions.

[1214] System Configuration

[1215] This system includes the following main components:

[1216] 1. Collection method: The server collects messages from multiple communication services (e.g., email, chat applications, social media) via a communication network. The collected messages are stored in a database.

[1217] 2. Analysis Method: The server analyzes the collected messages using natural language processing (NLP) techniques and generates summaries. An NLP engine such as the TextBlob library is used for natural language processing.

[1218] 3. Scoring method: The server uses a machine learning model (e.g., scikit-learn's logistic regression model) to score the importance of messages. Based on the importance score, messages are sorted in order of importance.

[1219] 4. Emotion Recognition Method: The server uses emotion recognition technology (e.g., TextBlob library) to recognize the emotions of the collected messages and assign emotion labels such as positive, negative, or neutral.

[1220] 5. Automatic generation method: The server uses the user's past reply data and sends automatically generated reply content based on the recognized sentiment to the user's terminal. This response content is generated based on templates or corpora.

[1221] 6. Modification and Submission Method: The user makes modifications to the automatically generated reply displayed on their device and then submits the final reply. The modified reply is sent via the API of the respective communication service (e.g., email or chat application).

[1222] Hardware and software configuration

[1223] Hardware: Smartphones, PCs, servers

[1224] software:

[1225] Collection method: Requests library

[1226] Analysis method: TextBlob library

[1227] Scoring method: scikit-learn library

[1228] Emotion recognition method: TextBlob library

[1229] Automatic generation method: Template engine (e.g., Jinja2)

[1230] Modification / Sending Method: smtplib library (for sending emails)

[1231] Specific example

[1232] For example, in a physical store, if a customer sends a message inquiring about product availability via various communication services, this system will operate as follows:

[1233] 1. The server collects all inquiry messages using APIs from services such as Instagram, LINE, and Facebook Messenger.

[1234] 2. Apply an NLP engine to the collected messages to generate a summary.

[1235] 3. Use emotion recognition technology to recognize the emotions in a message and assign emotion labels.

[1236] 4. An importance score is generated, and important messages are displayed at the top of the list.

[1237] 5. The server automatically generates a reply based on the user's past reply data and sentiment recognition results, and sends it to the user's terminal.

[1238] 6. The user reviews the reply on their device, makes any necessary corrections, confirms it, and sends it.

[1239] Example of a prompt

[1240] "Please create an appropriate template for automatically sending replies inquiring about product availability."

[1241] With the above configuration, the system of the present invention is capable of responding quickly and appropriately to customer inquiries. This is expected to improve customer satisfaction and the work efficiency of store employees.

[1242] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1243] Step 1:

[1244] The server collects messages from multiple communication services via a communication network. The server uses APIs to retrieve unread messages from services such as email, chat applications, and social media. The input is the API authentication token for each service, and the output is the retrieved message and its metadata (sender, recipient, timestamp, content, etc.). The collected messages are stored in a database.

[1245] Step 2:

[1246] This process applies natural language processing (NLP) techniques to messages collected by the server to generate summaries. An NLP engine (e.g., TextBlob library) is used to tokenize and parse the messages. The input is the collected message body, and the output is the generated summary. The process extracts key parts from the message to create a concise summary.

[1247] Step 3:

[1248] The server uses sentiment recognition technology to recognize the sentiment of the collected messages. It uses the TextBlob library to extract sentiment labels (positive, negative, neutral, etc.) from the text within the messages. The input is the summarized message body, and the output is a summary with sentiment labels added. Appropriate sentiment labels are added according to the tone of the message.

[1249] Step 4:

[1250] The server uses a machine learning model to score the importance of collected messages. It uses a scikit-learn logistic regression model to calculate importance scores based on input data (summaries, sentiment labels, and metadata). The input consists of sentiment-labeled summaries and metadata, and the output is the importance score. Based on the scoring results, the messages are sorted in order of importance.

[1251] Step 5:

[1252] The server uses the user's past reply data to generate automatically generated replies based on the perceived sentiment. A template engine (e.g., Jinja2) is used to create the automated replies. Inputs include importance scores, sorted messages, and the user's past reply patterns; output is the automatically generated reply. Contextually appropriate replies are generated based on templates.

[1253] Step 6:

[1254] The user reviews the automatically generated reply displayed on their device, makes any necessary corrections, and sends the final reply. The user checks the reply content using a smartphone or computer and makes corrections as needed. The input is the automatically generated reply, and the output is the corrected, final reply. After the user confirms and approves the reply, it is sent via the API of the respective communication service.

[1255] Through the processing steps described above, the server can efficiently execute a series of processes from message collection to automatic reply generation. For example, in response to a message inquiring about product inventory status, an automatic reply can be quickly generated, allowing store staff to modify and send it as needed.

[1256] Example prompt: "Please create an appropriate template for automatically sending a message inquiring about product availability."

[1257] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1258] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1259] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1260] [Fourth Embodiment]

[1261] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1262] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1263] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1264] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1265] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1266] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1267] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1268] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1269] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1270] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1271] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1272] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1273] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1274] This invention provides a system that automatically collects messages from multiple communication services used by a user, summarizes and evaluates their importance using natural language processing technology, and then provides the user with automatically generated replies. This system reduces the time and effort users spend checking and replying to messages.

[1275] System Configuration

[1276] This system includes the following main components:

[1277] 1. Collection method: The server collects messages from multiple communication services (e.g., email services, chat applications, social media) via a communication network.

[1278] 2. Analysis method: The server analyzes the collected messages using natural language processing technology and generates a summary.

[1279] 3. Scoring method: The server scores the importance of the analyzed messages and sorts them based on importance.

[1280] 4. Automatic generation method: The server uses the user's past reply data to automatically generate a reply and send it to the user's terminal.

[1281] 5. Correction and Submission Method: The user reviews the automatically generated reply displayed on their device, makes corrections as needed, and then submits the final reply.

[1282] Program processing

[1283] Collection methods

[1284] The server retrieves unread messages via APIs from Gmail, chat applications, and other services through the communication network. The server uses authentication tokens from each service to access user accounts and collect new and unread messages. The collected messages are stored in a database, which also stores metadata such as sender, recipient, and timestamp, in addition to the message content.

[1285] Analysis means

[1286] The server applies a natural language processing engine to the collected messages. This engine tokenizes the messages, tags them by part of speech, and analyzes the meaning of the sentences. As a result, a summary is generated and important information is extracted.

[1287] Scoring methods

[1288] The server uses a machine learning model to score the importance of the collected messages. This scoring takes into account factors such as the frequency of important keywords, the sender's role and relationship, and the urgency of past interactions. Based on the importance score, messages are sorted in order of importance.

[1289] automatic generation means

[1290] The server learns from the user's past reply data and creates automatically generated replies suitable for similar situations. These replies are sent to the user's device in JSON format, and the user can review them as drafts.

[1291] Correction / Transmission Methods

[1292] The terminal displays the automatically generated reply received from the server to the user. The user reviews the reply using the terminal's interface, makes any necessary corrections, and then confirms the final reply. The confirmed reply is sent to the server and then actually transmitted through the respective communication services.

[1293] Specific example

[1294] For example, if a user receives many unread emails or chat messages in a day, this system will work as follows:

[1295] 1. Collection method: The server uses the Gmail API and the chat app API to collect all unread messages.

[1296] 2. Analysis method: An NLP engine is applied to the collected messages to generate a summary.

[1297] 3. Scoring method: An importance score is calculated, and important messages are displayed at the top of the list.

[1298] 4. Automatic generation method: The server automatically generates a reply based on the user's past reply data and sends it to the user's terminal.

[1299] 5. Correction and Submission Method: The user reviews the reply on their device, makes any necessary corrections, confirms the message, and sends it.

[1300] This system significantly improves user work efficiency and reduces the time required to check and reply to messages.

[1301] The following describes the processing flow.

[1302] Step 1:

[1303] The server collects messages from communication services via the communication network. Specifically, it calls APIs of Gmail and chat applications to retrieve new and unread messages. In doing so, it uses API authentication tokens from each service to access the user's account.

[1304] Step 2:

[1305] The server stores the collected messages in a database. When saving, not only the message content but also metadata such as the sender, recipient, and timestamp are saved together.

[1306] Step 3:

[1307] The server initiates natural language processing on the stored messages. First, it tokenizes the messages, splitting them into individual words. Next, it tags each word with its part of speech and analyzes the meaning of the entire sentence. Based on this analysis, it generates a summary of the message.

[1308] Step 4:

[1309] The server uses a machine learning model to score importance based on the generated summary and message metadata. The scoring takes into account factors such as the frequency of important keywords, the relationship between the sender and receiver, and the urgency determined from past interactions.

[1310] Step 5:

[1311] The server sorts the scored messages by importance. Messages with higher scores are placed higher in the list, creating a list where important messages are displayed preferentially.

[1312] Step 6:

[1313] The server uses a model that has learned from the user's past reply data to create automatically generated replies. These replies are generated based on patterns in the user's past interactions and common reply templates.

[1314] Step 7:

[1315] The server sends the generated reply in JSON format to the user's device. The device receives this reply and displays it to the user.

[1316] Step 8:

[1317] The user reviews the automatically generated reply displayed on their device. They make any necessary corrections and finalize the reply.

[1318] Step 9:

[1319] The user submits their finalized reply by clicking the send button. The server then sends this reply through the API of the respective communication service (e.g., Gmail or chat application).

[1320] Through these steps, the system significantly reduces the time users spend checking and replying to messages, enabling more efficient communication.

[1321] (Example 1)

[1322] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1323] Currently, many users receive a large volume of messages daily through multiple communication services, resulting in the problem of significant time and effort being required to review and respond to them. Furthermore, the risk of overlooking important messages increases, highlighting the need for efficient message management. To address this challenge, a system is needed that can effectively automate message collection, summarization, importance assessment, and automatic responses.

[1324] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1325] In this invention, the server includes means for collecting messages from multiple communication services via a communication network; means for analyzing the collected messages using natural language processing technology and generating summaries; means for scoring the importance of the analyzed messages and sorting the messages based on importance; means for sending automatically generated reply content using the user's past reply data to the user terminal; means for the user to modify the automatically generated reply content displayed on the user terminal and send the final reply; means for storing metadata such as the sender, recipient, timestamp, and content of the collected messages; means for using a machine learning model to generate importance scores; means for tokenizing, tagging parts of speech, and analyzing sentences on the collected messages to extract important information; means for sending the automatically generated reply content in JSON format to the user terminal; and means for saving the reply content after user input and actually sending it via the communication service. As a result, users can significantly reduce the time required to check and reply to messages and efficiently manage important messages.

[1326] A "communication network" is the infrastructure used to send and receive data, and includes the internet and internal networks.

[1327] "Multiple communication services" refers to messaging services provided by different providers or applications, including, for example, email, chat apps, and social media.

[1328] A "message" is a unit of information that takes the form of text, images, audio, video, etc., and is sent from a sender to a recipient via a communication service.

[1329] "Natural language processing technology" refers to technologies for understanding, interpreting, and generating human language using computers, and includes processes such as tokenization, part-of-speech tagging, and semantic analysis.

[1330] A "summary" is information that extracts the most important information from the original message and presents it in a concise format.

[1331] "Importance" is a numerical value or rank that evaluates how important the content of a message is to the user.

[1332] "Scoring" is the process of evaluating the importance of a message and quantifying or ranking it.

[1333] "User's past reply data" refers to the history of reply messages previously sent by the user, and serves as reference data for creating automatically generated new reply content.

[1334] "Automatically generated reply content" refers to the content of a reply message that the system automatically creates based on the user's past reply data and other criteria.

[1335] A "user terminal" refers to a device such as a computer or smartphone used by a user, which can be used to check and reply to messages.

[1336] "Metadata" refers to information about the message itself, including attributes such as the sender, recipient, timestamp, and content.

[1337] A "machine learning model" is an algorithm or statistical model that uses large amounts of data to learn patterns and perform predictions or classifications.

[1338] "Tokenization" is the process of dividing text into meaningful units (tokens).

[1339] "Part-of-speech tagging" is the process of assigning a part of speech (noun, verb, adjective, etc.) to each token.

[1340] "Sentence analysis" is the process of analyzing the grammatical and semantic structure of a text.

[1341] JSON format is a lightweight data exchange format that represents data as key-value pairs.

[1342] This invention provides a system for efficiently managing messages by automatically collecting messages from multiple communication services used daily by users, and analyzing and organizing them using natural language processing technology. This system consists of a server and a user terminal and includes the following means.

[1343] System hardware and software

[1344] server

[1345] The server has the following functions:

[1346] A function that collects messages via a communication network. Specific examples include using the Gmail API or chat application APIs.

[1347] A function to store collected messages in a database. This uses a database management system such as MySQL.

[1348] A function that applies natural language processing technology. Google's BERT can be used.

[1349] A function that generates a summary based on the analysis results.

[1350] A feature that uses machine learning models to score importance and sort messages accordingly.

[1351] This feature automatically generates replies based on the user's past response data. It utilizes machine learning libraries such as TensorFlow and PyTorch for this purpose.

[1352] A function that sends automatically generated reply content to the user's terminal in JSON format.

[1353] User terminal

[1354] The user terminal has the following functions:

[1355] A function that displays automatically generated replies received from the server to the user.

[1356] A web application is used as an interface for users to review replies and make corrections as needed.

[1357] A function that sends the revised reply to the server and then finally sends the message through each communication service.

[1358] Data processing and data calculation

[1359] The server collects messages using the APIs of various communication services and stores them in a database. Metadata such as sender, recipient, timestamp, and content is also stored. Next, a natural language processing engine (such as BERT) is used to analyze the messages, extract important information, and generate summaries. Furthermore, a machine learning model is used to score the importance of the messages and sort them accordingly.

[1360] Based on the user's past reply data, an automatically generated reply is created and sent to the user's terminal in JSON format. The user reviews the content displayed on their terminal, makes any necessary corrections, and sends the final message to the server. The server then sends this message to the respective communication services, and the message is actually sent.

[1361] Specific example

[1362] For example, if a user receives many unread emails in their Gmail account, the server uses the Gmail API to collect these messages and store them in a database. A natural language processing engine is applied to the collected messages to generate summaries. Next, a machine learning model is used to calculate importance scores and sort the messages so that the most important ones are displayed first. The automatically generated reply is sent to the user's terminal in JSON format, and after the user reviews and modifies it, the final reply is sent to the communication service.

[1363] Example of a prompt

[1364] Examples of prompt statements to input into a generative AI model include:

[1365] "Collect unread messages received by users, generate summaries, assess their importance, and create prompts to generate automated replies."

[1366] The above describes the "modes for carrying out the invention" of the system related to the present invention. This system allows users to significantly reduce the time and effort required to check and reply to messages.

[1367] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1368] Step 1: Collecting Messages

[1369] Server operation:

[1370] Input: Use user authentication information (such as an OAuth token) to send requests to the APIs of various communication services (Gmail, chat apps, social media, etc.).

[1371] Data processing: Receive unread message data obtained from communication services and convert it to the required format.

[1372] Output: Content and metadata of unread messages (sender, recipient, timestamp, etc.).

[1373] Specific operation: The server sends a request to the Gmail API, retrieves unread messages, and stores them in the "messages" table in the database.

[1374] Step 2: Message Analysis

[1375] Server operation:

[1376] Input: Content and metadata of unread messages stored in the database.

[1377] Data processing: Apply a natural language processing engine (such as BERT) to tokenize messages, tag them by part of speech, and perform semantic analysis of the sentences.

[1378] Output: Message summary and extracted key information.

[1379] Specific operation: The server analyzes the message "Today's meeting is at 3 PM" and generates a summary such as "Meeting change."

[1380] Step 3: Importance scoring

[1381] Server operation:

[1382] Input: Summary and metadata of the analyzed message (sender's job title and relationship, urgency level, etc.).

[1383] Data processing: An importance score is calculated using a machine learning model. Scoring factors include keyword frequency and the urgency of past interactions.

[1384] Output: Importance score and a list of messages sorted by importance.

[1385] Specific operation: The server evaluates emails from superiors regarding "meeting changes" with a high score and places them higher in the list.

[1386] Step 4: Automatically generate reply content

[1387] Server operation:

[1388] Input: User's past reply data and a list of messages sorted by importance.

[1389] Data processing: Generate appropriate response content based on past response data. This is done using machine learning libraries (e.g., TensorFlow or PyTorch).

[1390] Output: Auto-generated reply content in JSON format.

[1391] Specific operation: The server generates a new "Understood" message based on the user's previous "Understood" reply and sends it to the user's terminal in JSON format.

[1392] Step 5: Presenting the response to the user

[1393] Device operation:

[1394] Input: Automated response content in JSON format received from the server.

[1395] Data processing: Convert the received JSON data into a user-friendly format and display it on the interface.

[1396] Output: The reply displayed to the user.

[1397] Specific action: The device displays an automatically generated reply message to the user saying "Understood."

[1398] Step 6: User modification and submission

[1399] User actions:

[1400] Input: The automatically generated reply content that was displayed.

[1401] Data processing: Modify the reply content as needed.

[1402] Output: The revised and final reply.

[1403] Specific action: The user changes "Understood" to "Understood. I have also confirmed the meeting time," and then presses the send button.

[1404] Step 7: Send the revised reply.

[1405] Server operation:

[1406] Input: The corrected reply received from the user's terminal.

[1407] Data processing: The modified response content is sent via the API of each communication service.

[1408] Output: The final reply message sent to each communication service.

[1409] Specific action: The server actually sends the corrected reply, "Understood. I've also confirmed the meeting time," via the Gmail API.

[1410] (Application Example 1)

[1411] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1412] In modern logistics centers, numerous employees exchange information using a variety of communication methods, resulting in a massive volume of messages, which poses a significant challenge. In this environment, quickly identifying and responding appropriately to important messages is crucial, but doing so manually is time-consuming and labor-intensive, hindering efficient operations. Furthermore, while automated generation of replies based on relevant past response data is desirable, effective methods for achieving this have lacked.

[1413] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1414] In this invention, the server includes means for collecting messages from multiple communication services via a communication network; means for analyzing the collected messages using natural language processing technology and generating summaries; means for scoring the importance of the analyzed messages and sorting the messages based on importance; means for sending automatically generated replies to a user terminal using the user's past reply data; means for the user to modify the automatically generated replies displayed on the user terminal and send the final reply; and means for automatically collecting and analyzing messages from multiple communication tools used by employees in the logistics center and automatically generating appropriate replies. This makes it possible to efficiently manage a vast number of messages in the logistics center, respond quickly to important messages, and improve work efficiency by automatically generating appropriate replies.

[1415] A "communication network" is the infrastructure used to connect computers to each other and to send and receive data.

[1416] "Communication services" refer to platforms for sending and receiving digital messages, such as email, chat applications, and social media.

[1417] "Message" refers to a unit of text, images, audio, and other data transmitted via electronic means of communication.

[1418] Natural language processing is a branch of computer science that analyzes and understands human language, and is often used for tasks such as text summarization and sentiment analysis.

[1419] A "summary" is a format in which the content of the original message is shortened and only the important information is extracted.

[1420] "Scoring" refers to the process of numerically evaluating the importance of a message, which makes it possible to prioritize messages.

[1421] A "user terminal" is a device used by a user to operate something, and includes smartphones, tablets, and personal computers.

[1422] "Reply data" refers to a history of messages previously sent by the user.

[1423] "Automatic generation" refers to the process of creating messages and replies based on pre-programmed algorithms, without human intervention.

[1424] A "logistics center" is a facility for storing, sorting, and shipping goods, and its purpose is to enable efficient logistics management.

[1425] "Employees" refers to workers who work at the logistics center and are responsible for tasks such as information gathering, sorting, shipping, and management.

[1426] "Communication tools" refer to applications and software used to send and receive messages, such as email, chat applications, and notifications from IoT devices.

[1427] This invention is a system for achieving efficient message management and rapid response within a logistics center. The system configuration and specific embodiments are described below.

[1428] System Configuration

[1429] The system includes the following main components:

[1430] 1. Server: Collects messages from multiple communication services via the communication network, and analyzes, summarizes, scores, and filters them using natural language processing technology. It also creates automatically generated replies based on the user's past reply data and sends them to the user's terminal.

[1431] 2. User terminal: Used by employees at the logistics center to view summaries of messages sent from the server, automatically generated replies, make corrections, and send final replies.

[1432] 3. Communication Network: This is the infrastructure that connects servers and user terminals and allows the use of multiple communication service APIs.

[1433] Hardware and software configuration

[1434] hardware

[1435] User devices: Smartphones, tablets, personal computers, etc.

[1436] Server: Responsible for data processing as the central management system.

[1437] software

[1438] Python: Used for implementing natural language processing and machine learning models.

[1439] Generative AI models: GPT-3, BERT, etc., used for text generation and information extraction.

[1440] APIs from various communication services, such as the Gmail API, Slack API, and IoT device APIs, are used to collect messages.

[1441] Database: Used for managing messages and metadata.

[1442] Processing flow

[1443] The server first collects new and unread messages through the APIs of each communication service. Next, it applies a natural language processing engine (e.g., SpaCy) to the collected messages to generate summaries. Then, it uses a machine learning model (e.g., importance scoring using TF-IDF) to score the importance of the messages and sorts them in order of importance.

[1444] Subsequently, the server automatically generates appropriate replies using a generative AI model (e.g., GPT-3) based on the user's past reply data. This generated reply is sent to the user's terminal in JSON format, allowing the user to review, revise, and submit their final reply as a draft.

[1445] Specific example

[1446] For example, if an employee at a logistics center receives a large number of unread messages and notifications, this system will work as follows:

[1447] 1. Message Collection and Analysis: The server uses the Gmail API, Slack API, and IoT device APIs to collect all unread messages and notifications, and generates summaries using a natural language processing engine.

[1448] 2. Scoring and Sorting: Use a machine learning model to calculate importance scores and prioritize and sort important messages.

[1449] 3. Automatic generation of replies: The server uses an AI model to automatically generate appropriate replies based on the collected messages.

[1450] 4. Review and correction on the user's device: The user reviews the reply on their device, makes any necessary corrections, and then confirms and sends the final reply.

[1451] Examples of prompts for generative AI models

[1452] example:

[1453] Please generate a response appropriate to the following situation:

[1454] situation:

[1455] We received an inquiry from a customer stating that their ordered item has not yet arrived. Upon checking the tracking information, we found that the delivery is delayed.

[1456] reply:

[1457] This system streamlines message management within the logistics center, significantly improving employee work efficiency. Furthermore, it enables quick responses to important messages, contributing to increased customer satisfaction.

[1458] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1459] Step 1:

[1460] The server collects messages from multiple communication services (such as Gmail, Slack, and IoT devices) via a communication network. In this step, it uses APIs to retrieve new and unread messages and saves them to a local database. The input is the message data obtained from each communication service, and the output is the message data stored in the local database.

[1461] Step 2:

[1462] The server applies natural language processing techniques to the collected messages to generate summaries. In this step, libraries such as SpaCy are used to tokenize the messages, extract important information, and generate summaries. The input is message data obtained from a local database, and the output is the generated summary data. Specifically, the first few sentences of the message body are extracted and used as the summary.

[1463] Step 3:

[1464] The server scores and sorts the importance of the analyzed messages. In this step, a machine learning model (e.g., TF-IDF scoring) is used to evaluate the importance of the messages. The input is summarized message data, and the output is a list of messages that have been scored and sorted by importance. Specifically, the TF-IDF score for each message is calculated, and the messages are sorted based on that score.

[1465] Step 4:

[1466] The server uses the user's past reply data to create automatically generated replies using a generative AI model (e.g., GPT-3). In this step, the reply is generated based on the prompt. The input is scored message data and the user's past reply data, and the output is the generated reply. Specifically, the prompt is input to the generative AI model, and the generated reply is retrieved.

[1467] Step 5:

[1468] The server automatically generates a reply and sends it to the user's terminal. In this step, the reply generated in JSON format is transferred to the user's terminal. The input is the generated reply, and the output is a notification that the transmission to the user's terminal is complete. Specifically, the reply is sent using the user's terminal's identification information.

[1469] Step 6:

[1470] The user terminal displays the automatically generated reply to the user. In this step, the reply is displayed on the interface, and the user is provided with the ability to make modifications. The input is the automatically generated reply sent from the server, and the output is the reply that has been reviewed and modified by the user. Specifically, the reply is rendered on the display interface.

[1471] Step 7:

[1472] The user makes final revisions to their reply and sends it. In this step, the user reviews and confirms the revised reply, and then sends it via the server through the respective communication services. The input is the reply revised by the user, and the output is the final reply message sent. Specifically, based on the user's confirmation action, the message is sent through the respective communication service APIs.

[1473] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1474] This invention is a system that automatically collects messages from multiple communication services used by a user, analyzes them using natural language processing technology and an emotion engine to summarize and evaluate their importance, and provides the user with automatically generated replies. This system reduces the time and effort users spend checking and replying to messages, and also enables appropriate responses that take into account the user's emotions.

[1475] System Configuration

[1476] This system includes the following main components:

[1477] 1. Collection method: The server collects messages from multiple communication services (email services, chat applications, social media) via a communication network.

[1478] 2. Analysis method: The server analyzes the collected messages using natural language processing technology and generates a summary.

[1479] 3. Scoring method: The server scores the importance of the analyzed messages and sorts them based on importance.

[1480] 4. Emotion Recognition Method: The server uses an emotion engine to recognize emotions from collected messages.

[1481] 5. Automatic generation means: The server automatically generates a reply based on the user's past reply data and recognized emotions and sends it to the user's terminal.

[1482] 6. Correction and Submission Method: The user reviews the automatically generated reply displayed on their device, makes corrections as needed, and then submits the final reply.

[1483] Program processing

[1484] Collection methods

[1485] The server retrieves unread messages via APIs from services such as Gmail and chat applications through the communication network. It accesses user accounts using API authentication tokens for each service to collect new and unread messages. The collected messages are stored in a database, which stores not only the message content but also metadata such as sender, recipient, and timestamp.

[1486] Analysis means

[1487] The server applies a natural language processing engine to the collected messages. First, the messages are tokenized and split into individual words. Next, the part of speech of each word is identified, the meaning of the sentence is analyzed, and a message summary is generated.

[1488] emotion recognition means

[1489] The server uses an emotion engine to recognize the sentiment of a message. The emotion engine extracts emotions from the text within the message and assigns sentiment labels such as positive, negative, and neutral. The recognized sentiment influences importance scoring and automated reply generation.

[1490] Scoring methods

[1491] The server uses a machine learning model to score the importance of the collected messages. The scoring considers factors such as the frequency of important keywords, the sender's role and relationship, and the results of sentiment recognition. Based on the importance score, messages are sorted in order of importance.

[1492] automatic generation means

[1493] The server automatically generates a response based on the user's past reply data and perceived emotions. This response is generated based on patterns of past interactions and common reply templates. In particular, it generates a response with an appropriate tone and content that takes the results of emotion recognition into account.

[1494] Correction / Transmission Methods

[1495] The device displays the automatically generated reply received from the server to the user. The user reviews the reply on the device, makes any necessary corrections, and then confirms the final reply. The confirmed reply is sent to the server and then actually transmitted through the APIs of each communication service (e.g., Gmail or chat applications).

[1496] Specific example

[1497] For example, if a user receives many unread emails or chat messages in a day, this system will work as follows:

[1498] 1. Collection method: The server uses the Gmail API and the chat app API to collect all unread messages.

[1499] 2. Analysis method: An NLP engine is applied to the collected messages to generate a summary.

[1500] 3. Emotion Recognition Method: The emotion engine recognizes the emotion of the message and assigns an emotion label.

[1501] 4. Scoring method: An importance score is calculated, and important messages are displayed at the top of the list.

[1502] 5. Automatic generation method: The server automatically generates a reply based on the user's past reply data and sentiment recognition results, and sends it to the user's terminal.

[1503] 6. Correction and Submission Method: The user reviews the reply on their device, makes any necessary corrections, confirms the message, and sends it.

[1504] This system significantly improves user productivity, reducing the time required to check and reply to messages, and enabling appropriate responses that take user emotions into consideration.

[1505] The following describes the processing flow.

[1506] Step 1:

[1507] The server collects messages from communication services via the communication network. Specifically, it calls APIs of Gmail and chat applications to retrieve new and unread messages. In doing so, it uses API authentication tokens from each service to access the user's account.

[1508] Step 2:

[1509] The server stores the collected messages in a database. When saving, not only the message content but also metadata such as the sender, recipient, and timestamp are saved together.

[1510] Step 3:

[1511] The server initiates natural language processing (NLP) on the stored messages. First, the messages are tokenized and split into individual words. Next, part-of-speech tags are added to each word, and the meaning of the entire sentence is analyzed to generate a summary of the message.

[1512] Step 4:

[1513] The server uses an emotion engine to recognize the emotion of a message. The emotion engine extracts emotion labels such as positive, negative, and neutral from the message and adds this emotion information to the message data.

[1514] Step 5:

[1515] The server uses a machine learning model to score the importance of the collected messages. The scoring considers factors such as the frequency of important keywords, the sender's role and relationship, and the results of sentiment recognition. Based on the importance score, messages are sorted in order of importance.

[1516] Step 6:

[1517] The server automatically generates replies based on the user's past reply data and sentiment recognition results. The generated replies are based on patterns of past interactions and common reply templates, and include replies with an appropriate tone and content, taking into account the sentiment recognition results in particular.

[1518] Step 7:

[1519] The server sends the generated reply in JSON format to the user's device. The device receives this reply and displays it to the user.

[1520] Step 8:

[1521] The user reviews the automatically generated reply displayed on their device. They make any necessary corrections and finalize the reply.

[1522] Step 9:

[1523] The user submits their finalized reply by clicking the send button. The server then sends this reply through the API of the respective communication service (e.g., Gmail or chat application).

[1524] Through this series of steps, users can significantly reduce the time spent checking and responding to messages, and also be able to provide appropriate responses that take into account the user's emotions.

[1525] (Example 2)

[1526] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1527] In modern society, users utilize multiple communication services (email, chat applications, social media, etc.) and receive a large volume of messages daily. However, responding quickly and appropriately to all messages is difficult, and managing and replying to messages consumes a significant amount of time and effort. Furthermore, accurately understanding the content, importance, and sentiment of a message, and responding accordingly, requires high skill. Therefore, there is a need for technology that can quickly generate appropriate responses and reduce the user's time and effort.

[1528] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting messages from multiple communication services via a communication network, means for storing metadata (sender, recipient, timestamp, content) of the collected messages, means for tokenizing the collected messages using natural language processing technology, assigning part-of-speech tags, and generating summaries, means for recognizing the sentiment of messages using a sentiment engine and labeling the results, means for scoring the importance of messages analyzed using a machine learning model and sorting messages based on importance, means for sending automatically generated reply content based on the user's past reply data and recognized sentiment to the user terminal, and means for the user to modify the automatically generated reply content displayed on the user terminal and send the final reply. This enables the user to respond quickly and appropriately to a large volume of messages, as well as significantly reduce the user's time and effort.

[1529] A "communication network" refers to a computer network used for sending and receiving data.

[1530] "Communication services" refer to services that allow users to exchange information, such as email, chat applications, and social media.

[1531] A "message" refers to information such as text, images, and files sent by a user.

[1532] "Metadata" refers to supplementary information accompanying a message, including attribute data such as sender, recipient, timestamp, and content.

[1533] "Natural language processing technology" refers to the technology used to process natural language used by humans using computers.

[1534] "Tokenization" refers to the process of dividing a continuous text into words or phrases.

[1535] "Part-of-speech tagging" refers to the process of identifying which part of speech each tokenized word belongs to, such as a noun, verb, or adjective.

[1536] A "summary" refers to a text that extracts the main points of the original message and presents them concisely.

[1537] An "emotion engine" refers to an algorithm that recognizes and labels emotions from text data.

[1538] A "machine learning model" refers to an algorithm that learns patterns from data and uses those learning results to analyze and predict new data.

[1539] "Importance scoring" refers to the process of numerically evaluating the importance of a message.

[1540] "Sorting" refers to the process of ordering messages based on their importance score.

[1541] "Automatically generated replies" refer to reply messages created by a computer based on the user's past reply patterns and perceived emotions.

[1542] "User terminal" refers to a device (such as a smartphone, tablet, or personal computer) that a user uses to access the system.

[1543] "Correction" refers to the act of a user making changes or additions to automatically generated replies.

[1544] "Final reply" refers to the reply message that the user has finalized after making revisions.

[1545] This invention is a system that automatically collects messages from multiple communication services used by a user, analyzes them using natural language processing technology and an emotion engine to perform summarization and importance evaluation, and provides the user with automatically generated replies.

[1546] This system includes the following main components:

[1547] 1. Collection Method: The server collects messages from multiple communication services (email services, chat applications, social media) via the communication network. Specifically, it uses the Gmail API and chat application APIs to retrieve users' unread messages. It accesses user accounts using API authentication tokens for each service to collect new and unread messages. The collected messages are stored in a database, and in addition to the message content, metadata such as sender, recipient, and timestamp is also stored.

[1548] 2. Analysis Method: The server tokenizes the collected messages using natural language processing techniques (e.g., SpaCy, NLTK) and tags them by part of speech. Tokenization divides the message text into words, and part-of-speech tagging identifies the part of speech of each word. Next, the server performs semantic analysis of the sentences and generates a summary of the messages. The NLTK summarization algorithm is used for this purpose.

[1549] 3. Emotion Recognition Method: The server uses an emotion engine (e.g., Hugging Face's Transformers model) to recognize the emotion of a message. The emotion engine extracts emotions from the text within the message and assigns emotion labels such as positive, negative, and neutral. The recognized emotion influences importance scoring and automatic reply generation.

[1550] 4. Scoring Method: The server uses a machine learning model (e.g., a classification model from Scikit-learn) to score the importance of the collected messages. The scoring considers factors such as the frequency of important keywords, the sender's role and relationship, and the results of sentiment recognition. Based on the importance score, messages are sorted in order of importance.

[1551] 5. Automatic generation method: The server automatically generates response content based on the user's past response data and recognized emotions. This response content is generated based on patterns of past interactions and general response templates. In particular, responses with appropriate tone and content are generated, taking into account the results of emotion recognition. The server generates responses using a generation AI model (e.g., GPT-3).

[1552] 6. Correction and Sending Method: The terminal displays the automatically generated reply received from the server to the user. The user reviews the reply on the terminal, makes any necessary corrections, and then confirms the final reply. The confirmed reply is sent to the server and actually sent through the APIs of each communication service (e.g., Gmail or chat applications).

[1553] As a concrete example, if a user receives many unread emails and chat messages in a day, this system will work as follows:

[1554] 1. The server uses the Gmail API and the chat app API to collect all unread messages.

[1555] 2. A natural language processing engine is applied to the collected messages to generate a summary.

[1556] 3. The emotion engine recognizes the emotion of the message and assigns an emotion label.

[1557] 4. An importance score is calculated, and important messages are displayed at the top of the list.

[1558] 5. The server automatically generates a reply based on the user's past reply data and sentiment recognition results, and sends it to the user's terminal.

[1559] 6. The user reviews the reply on their device, makes any necessary corrections, confirms it, and sends it.

[1560] Examples of input prompts for a generative AI model are shown below:

[1561] 1. "Please describe the process for automatically collecting unread messages from email and chat applications, generating summaries, sorting them based on importance scores, and automatically generating appropriate replies."

[1562] 2. "Explain how to use an emotion recognition engine to label the emotions of messages and then use that information to automatically generate replies."

[1563] These prompts allow the user to get guidance on the detailed operation and implementation of the system.

[1564] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1565] Program processing flow

[1566] Step 1:

[1567] The server collects unread messages from multiple communication services (e.g., email services, chat applications, social media) via a communication network. The input requires API authentication tokens and user IDs for each communication service. The server uses these to call each API and retrieve new and unread messages. The output includes metadata such as message content, sender, recipient, and timestamp, which are stored in a database.

[1568] Specific actions:

[1569] The server calls the Gmail API to retrieve the user's unread emails.

[1570] The server uses the chat application API to collect unread messages.

[1571] Step 2:

[1572] The server uses a natural language processing engine (e.g., SpaCy) to tokenize the messages it collects. The message text is required as input. The server divides the message text into words and outputs a sequence of tokenized words.

[1573] Specific actions:

[1574] The server uses SpaCy's nlp(text) function to tokenize the message text.

[1575] Step 3:

[1576] The server performs part-of-speech tagging on tokenized words. The input is a sequence of tokenized words. The server assigns a part-of-speech tag to each word, and the output is a sequence of words tagged with part-of-speech.

[1577] Specific actions:

[1578] The part-of-speech information of the tokens generated by SpaCy is extracted, and tags are assigned to each token.

[1579] Step 4:

[1580] The server performs semantic analysis of the text and generates a message summary. The input requires a sequence of words tagged with parts of speech. The server uses natural language processing techniques (e.g., NLTK) to extract key points and obtains a summary as output.

[1581] Specific actions:

[1582] The server uses the NLTK summarization algorithm to extract important sentences.

[1583] Step 5:

[1584] The server uses an emotion engine (e.g., Hugging Face's Transformers model) to recognize emotions in the collected messages. The message text is required as input. The emotion engine parses the text and outputs emotion labels such as positive, negative, or neutral.

[1585] Specific actions:

[1586] The server calls pipeline('sentiment-analysis') to retrieve sentiment labels from the message text.

[1587] Step 6:

[1588] The server uses a machine learning model (e.g., a classification model in Scikit-learn) to score the importance of collected messages. Input includes sentiment labels, important keywords, and sender information. The server calculates a score based on this information, and the output is the importance score.

[1589] Specific actions:

[1590] We will use a classification model from Scikit-learn to calculate an importance score for each message.

[1591] Step 7:

[1592] The server sorts messages based on their importance score. The input is a list of scored messages. The server sorts the messages by importance and outputs a sorted list of messages.

[1593] Specific actions:

[1594] The server sorts the message list based on its importance score, placing the most important messages at the top of the list.

[1595] Step 8:

[1596] The server automatically generates response content based on the user's past response data and recognized sentiment. Past response data and sentiment labels are required as input. A generative AI model (e.g., GPT-3) is used to generate a response based on the prompt, and the response content is obtained as output.

[1597] Specific actions:

[1598] This function calls the GPT-3 API and generates a response based on the prompt.

[1599] Input prompt example: "Generate an appropriate reply for the unread message."

[1600] Step 9:

[1601] The terminal displays the automatically generated reply received from the server to the user. The generated reply is required as input. The user reviews the reply on the terminal, makes any necessary corrections, and then receives the final confirmed reply as output.

[1602] Specific actions:

[1603] The user reviews and modifies the reply displayed on their device.

[1604] Step 10:

[1605] The user sends their confirmed reply to the server, which then uses the APIs of the respective communication services to actually send the final reply. The input requires the user's confirmed and revised final reply. The server sends the reply via the API and notifies the user when the transmission is complete.

[1606] Specific actions:

[1607] The server calls the Gmail API to send the email.

[1608] Send a message using the chat application API.

[1609] (Application Example 2)

[1610] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1611] In modern brick-and-mortar stores, customer inquiries are increasingly pouring in through various communication services. This means that store employees have to spend a considerable amount of time and effort reviewing and responding to messages appropriately. Furthermore, failure to respond immediately can lead to decreased customer satisfaction. Therefore, there is a need for technology that can efficiently manage messages and automatically generate appropriate responses.

[1612] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting messages from multiple communication services via a communication network, means for analyzing the collected messages using natural language processing technology and generating summaries, means for scoring the importance of the analyzed messages and sorting the messages based on importance, means for sending automatically generated reply content using the user's past reply data to the user terminal, means for the user to modify the automatically generated reply content displayed on the user terminal and send the final reply, means for applying emotion recognition technology to the collected messages and assigning emotion labels, and means for generating automatic reply content to respond to customer inquiry messages in a physical store, which is then modified and sent by the store staff. This makes it possible to respond to customer inquiries quickly and appropriately.

[1613] A "communication network" is an infrastructure for exchanging information between different locations, such as the internet or mobile phone networks.

[1614] A "message" is a unit of information that is sent and received in various formats, such as text, images, and videos.

[1615] "Natural language processing technology" is a technology that uses computers to analyze and understand human language.

[1616] A "means of generating summaries" is a technique for concisely summarizing the main content and key points of collected messages.

[1617] "Scoring importance" is the process of quantifying the importance of a message based on the content and sender information collected from that message.

[1618] "Emotion recognition technology" is a technology that detects emotions from data such as text and audio, and assigns specific emotion labels (positive, negative, neutral, etc.) to them.

[1619] "Automatically generated replies" are replies that a computer automatically creates based on the user's past reply patterns and templates.

[1620] A "user terminal" is a device (such as a smartphone or personal computer) that a user uses to receive or send information.

[1621] "A means of making revisions and sending a final reply" refers to the process by which a user reviews an automatically generated reply, makes changes as needed, and then sends the message.

[1622] A "physical store" is a store that exists physically and is a place where commercial transactions are conducted directly with customers.

[1623] An "inquiry message" is a message sent by a customer to a store or service provider with questions or requests.

[1624] This invention provides a system that enables users to efficiently manage messages from multiple communication services and provide prompt and appropriate responses. This invention significantly reduces the time and effort required to check and reply to messages. Furthermore, it enables appropriate responses that take into account the user's emotions.

[1625] System Configuration

[1626] This system includes the following main components:

[1627] 1. Collection method: The server collects messages from multiple communication services (e.g., email, chat applications, social media) via a communication network. The collected messages are stored in a database.

[1628] 2. Analysis Method: The server analyzes the collected messages using natural language processing (NLP) techniques and generates summaries. An NLP engine such as the TextBlob library is used for natural language processing.

[1629] 3. Scoring method: The server uses a machine learning model (e.g., scikit-learn's logistic regression model) to score the importance of messages. Based on the importance score, messages are sorted in order of importance.

[1630] 4. Emotion Recognition Method: The server uses emotion recognition technology (e.g., TextBlob library) to recognize the emotions of the collected messages and assign emotion labels such as positive, negative, or neutral.

[1631] 5. Automatic generation method: The server uses the user's past reply data and sends automatically generated reply content based on the recognized sentiment to the user's terminal. This response content is generated based on templates or corpora.

[1632] 6. Modification and Submission Method: The user makes modifications to the automatically generated reply displayed on their device and then submits the final reply. The modified reply is sent via the API of the respective communication service (e.g., email or chat application).

[1633] Hardware and software configuration

[1634] Hardware: Smartphones, PCs, servers

[1635] software:

[1636] Collection method: Requests library

[1637] Analysis method: TextBlob library

[1638] Scoring method: scikit-learn library

[1639] Emotion recognition method: TextBlob library

[1640] Automatic generation method: Template engine (e.g., Jinja2)

[1641] Modification / Sending Method: smtplib library (for sending emails)

[1642] Specific example

[1643] For example, in a physical store, if a customer sends a message inquiring about product availability via various communication services, this system will operate as follows:

[1644] 1. The server collects all inquiry messages using APIs from services such as Instagram, LINE, and Facebook Messenger.

[1645] 2. Apply an NLP engine to the collected messages to generate a summary.

[1646] 3. Use emotion recognition technology to recognize the emotions in a message and assign emotion labels.

[1647] 4. An importance score is generated, and important messages are displayed at the top of the list.

[1648] 5. The server automatically generates a reply based on the user's past reply data and sentiment recognition results, and sends it to the user's terminal.

[1649] 6. The user reviews the reply on their device, makes any necessary corrections, confirms it, and sends it.

[1650] Example of a prompt

[1651] "Please create an appropriate template for automatically sending replies inquiring about product availability."

[1652] With the above configuration, the system of the present invention is capable of responding quickly and appropriately to customer inquiries. This is expected to improve customer satisfaction and the work efficiency of store employees.

[1653] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1654] Step 1:

[1655] The server collects messages from multiple communication services via a communication network. The server uses APIs to retrieve unread messages from services such as email, chat applications, and social media. The input is the API authentication token for each service, and the output is the retrieved message and its metadata (sender, recipient, timestamp, content, etc.). The collected messages are stored in a database.

[1656] Step 2:

[1657] This process applies natural language processing (NLP) techniques to messages collected by the server to generate summaries. An NLP engine (e.g., TextBlob library) is used to tokenize and parse the messages. The input is the collected message body, and the output is the generated summary. The process extracts key parts from the message to create a concise summary.

[1658] Step 3:

[1659] The server uses sentiment recognition technology to recognize the sentiment of the collected messages. It uses the TextBlob library to extract sentiment labels (positive, negative, neutral, etc.) from the text within the messages. The input is the summarized message body, and the output is a summary with sentiment labels added. Appropriate sentiment labels are added according to the tone of the message.

[1660] Step 4:

[1661] The server uses a machine learning model to score the importance of collected messages. It uses a scikit-learn logistic regression model to calculate importance scores based on input data (summaries, sentiment labels, and metadata). The input consists of sentiment-labeled summaries and metadata, and the output is the importance score. Based on the scoring results, the messages are sorted in order of importance.

[1662] Step 5:

[1663] The server uses the user's past reply data to generate automatically generated replies based on the perceived sentiment. A template engine (e.g., Jinja2) is used to create the automated replies. Inputs include importance scores, sorted messages, and the user's past reply patterns; output is the automatically generated reply. Contextually appropriate replies are generated based on templates.

[1664] Step 6:

[1665] The user reviews the automatically generated reply displayed on their device, makes any necessary corrections, and sends the final reply. The user checks the reply content using a smartphone or computer and makes corrections as needed. The input is the automatically generated reply, and the output is the corrected, final reply. After the user confirms and approves the reply, it is sent via the API of the respective communication service.

[1666] Through the processing steps described above, the server can efficiently execute a series of processes from message collection to automatic reply generation. For example, in response to a message inquiring about product inventory status, an automatic reply can be quickly generated, allowing store staff to modify and send it as needed.

[1667] Example prompt: "Please create an appropriate template for automatically sending a message inquiring about product availability."

[1668] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1669] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1670] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1671] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1672] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1673] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1674] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1675] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1676] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1677] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1678] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1679] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1680] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1681] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1682] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1683] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1684] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1685] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1686] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1687] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1688] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1689] The following is further disclosed regarding the embodiments described above.

[1690] (Claim 1)

[1691] A means of collecting messages from multiple communication services through a communication network,

[1692] A means for analyzing collected messages using natural language processing technology and generating summaries,

[1693] A means for scoring the importance of the analyzed messages and sorting the messages based on their importance,

[1694] A means of sending automatically generated reply content to the user's terminal using the user's past reply data,

[1695] A means by which the user can modify the automatically generated reply content displayed on the user's terminal and send the final reply,

[1696] A system that includes this.

[1697] (Claim 2)

[1698] The system according to claim 1, further comprising means for storing metadata such as the sender, recipient, timestamp, and content of collected messages.

[1699] (Claim 3)

[1700] The system according to claim 1, further comprising means for using a machine learning model to generate importance scores.

[1701] "Example 1"

[1702] (Claim 1)

[1703] A means of collecting messages from multiple communication services through a communication network,

[1704] A means for analyzing collected messages using natural language processing technology and generating summaries,

[1705] A means for scoring the importance of the analyzed messages and sorting the messages based on their importance,

[1706] A means of sending automatically generated reply content to the user's terminal using the user's past reply data,

[1707] A means by which the user can modify the automatically generated reply content displayed on the user's terminal and send the final reply,

[1708] A system that includes this.

[1709] (Claim 2)

[1710] The system according to claim 1, further comprising means for storing metadata such as the sender, recipient, timestamp, and content of collected messages.

[1711] (Claim 3)

[1712] The system according to claim 1, further comprising means for using a machine learning model to generate importance scores.

[1713] (Claim 4)

[1714] The system according to claim 1, further comprising means for tokenizing, part-of-speech tagging, and sentence analysis of collected messages to extract important information.

[1715] (Claim 5)

[1716] The system according to claim 1, further comprising means for sending automatically generated reply content in JSON format to a user terminal.

[1717] (Claim 6)

[1718] The system according to claim 1, further comprising means for saving the user's modified reply and actually transmitting it through a communication service.

[1719] "Application Example 1"

[1720] (Claim 1)

[1721] A means of collecting messages from multiple communication services through a communication network,

[1722] A means for analyzing collected messages using natural language processing technology and generating summaries,

[1723] A means for scoring the importance of the analyzed messages and sorting the messages based on their importance,

[1724] A means of sending automatically generated reply content to the user's terminal using the user's past reply data,

[1725] A means by which the user can modify the automatically generated reply content displayed on the user's terminal and send the final reply,

[1726] A means to automatically collect and analyze messages from multiple communication tools used by employees within a logistics center, and to automatically generate appropriate response content.

[1727] A system that includes this.

[1728] (Claim 2)

[1729] The system according to claim 1, further comprising means for storing metadata such as the sender, recipient, timestamp, and content of collected messages.

[1730] (Claim 3)

[1731] The system according to claim 1, further comprising means for using a machine learning model to generate importance scores.

[1732] "Example 2 of combining an emotion engine"

[1733] (Claim 1)

[1734] A means of collecting messages from multiple communication services through a communication network,

[1735] A means of storing the metadata of the collected messages (sender, recipient, timestamp, content),

[1736] A means for tokenizing collected messages using natural language processing technology, assigning part-of-speech tags, and generating summaries,

[1737] A means of recognizing the emotion of a message using an emotion engine and labeling the result,

[1738] A means of scoring the importance of messages analyzed using a machine learning model and sorting the messages based on their importance,

[1739] A means for sending automatically generated reply content to the user's terminal based on the user's past reply data and recognized emotions,

[1740] A means by which the user can modify the automatically generated reply content displayed on the user's terminal and send the final reply,

[1741] A system that includes this.

[1742] (Claim 2)

[1743] The system according to claim 1, further comprising means for analyzing messages and recognizing emotions in cooperation with a natural language processing engine and an emotion engine.

[1744] (Claim 3)

[1745] The system according to claim 1, further comprising means for automatically generating reply content based on past reply patterns and recognized emotions using a generative AI model.

[1746] "Application example 2 when combining with an emotional engine"

[1747] (Claim 1)

[1748] A means of collecting messages from multiple communication services through a communication network,

[1749] A means for analyzing collected messages using natural language processing technology and generating summaries,

[1750] A means for scoring the importance of the analyzed messages and sorting the messages based on their importance,

[1751] A means of sending automatically generated reply content to the user's terminal using the user's past reply data,

[1752] A means by which the user can modify the automatically generated reply content displayed on the user's terminal and send the final reply,

[1753] A means of applying emotion recognition technology to collected messages and assigning emotion labels,

[1754] A method for generating automated replies to customer inquiries in physical stores, which are then edited and sent by store staff.

[1755] A system that includes this.

[1756] (Claim 2)

[1757] The system according to claim 1, further comprising means for storing metadata such as the sender, recipient, timestamp, and content of collected messages.

[1758] (Claim 3)

[1759] The system according to claim 1, further comprising means for using a machine learning model to generate importance scores. [Explanation of Symbols]

[1760] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of collecting messages from multiple communication services through a communication network, A means for analyzing collected messages using natural language processing technology and generating summaries, A means for scoring the importance of the analyzed messages and sorting the messages based on their importance, A means of sending automatically generated reply content to the user's terminal using the user's past reply data, A means by which the user can modify the automatically generated reply content displayed on the user's terminal and send the final reply, A system that includes this.

2. The system according to claim 1, further comprising means for storing metadata such as the sender, recipient, timestamp, and content of collected messages.

3. The system according to claim 1, further comprising means for using a machine learning model to generate importance scores.

Images