System

Abstract

A system is provided.SOLUTION: The system includes a means for acquiring history data from various communication tools, a means for converting the history data into a unified format, a means for storing the data converted into the unified format in a database, a means for retrieving corresponding history data from the database on the basis of retrieval conditions designated by a user, and a means for visualizing the retrieved history data to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Because correspondence history is scattered across various communication tools (email, chat, etc.) within an organization, it is difficult to quickly and efficiently search and reference specific information. Furthermore, if the correspondence history of past employees is also included in the search, the complexity and effort increases. To solve these problems, a system is needed that aggregates the history in a unified format and makes it easy to search. [Means for solving the problem]

[0005] The present invention provides a means for acquiring history data from various communication tools, converting the data into a unified format, and storing it in a database. Furthermore, a system is constructed that searches the database for corresponding history data based on search criteria specified by the user and visualizes the data for the user. The system of the present invention has a means for sorting search results in chronological order or by relevance, and also includes a means for processing search requests from multiple users in parallel, enabling efficient and rapid information retrieval.

[0006] "Historical Data" means records of past communications, including messages sent and received via email, chat, and other communications tools, files, and associated metadata.

[0007] "Communication tools" refers to software or hardware that enables users to exchange information, such as email, instant messaging, video conferencing, audio conferencing, social networking sites, and collaboration platforms.

[0008] A "uniform format" is a standardized data format for converting historical data recorded in different formats into a common data structure or format.

[0009] A "database" is an information system designed to efficiently store, manage, and retrieve large amounts of data.

[0010] "Search criteria" means keywords, date ranges, senders, recipients, or other filtering criteria that a user specifies to retrieve specific information.

[0011] "Visualization" refers to displaying data in a format that is easy for a user to understand, and includes formats such as text, graphs, charts, and tables.

[0012] "Chronological order" means arranging data in the order in which they occurred, from the oldest information to the most recent information.

[0013] "Relevance order" refers to a method of sorting search results in order of relevance to the user's search criteria.

[0014] "Parallel processing" refers to the execution of multiple processes simultaneously, including the ability to process search requests from multiple users simultaneously. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention provides a specific method for implementing a system that acquires historical data from various communication tools, converts it into a unified format, stores it in a database, searches for corresponding historical data based on search conditions specified by the user, and visualizes it for the user.

[0037] Data collection

[0038] server:

[0039] 1. The server periodically retrieves historical data from each communication tool (e.g., mail server, chat server). This includes retrieving new emails from the mail server using the IMAP or POP3 protocol, and retrieving messages using the chat tool's API.

[0040] 2. A parser is used to convert the retrieved data into a unified format. For example, for email data, information such as the subject, body, sender, recipient, and date and time is retrieved, and for chat data, message content, sender, recipient, and timestamp information is retrieved.

[0041] 3. The converted data is saved in the database. This data is checked for duplicates and saved in the database in a unified format.

[0042] Data Search

[0043] User:

[0044] 1. The user accesses a dedicated search interface and enters search criteria such as search keywords, a date range, or specific senders / receivers. The interface is presented using text boxes, drop-down menus, and calendar widgets.

[0045] 2. Set the search conditions and press the search button.

[0046] server:

[0047] 1. The server receives a search request from the user and generates a SQL query based on the criteria. For example, if you specify the keyword "Project X" and a specific date range, it generates a SQL statement to search for historical data that matches those criteria.

[0048] 2. The generated SQL query is executed against the database to retrieve search results, which are then sorted by chronological order and relevance.

[0049] Display and Operation

[0050] Device:

[0051] 1. The search results returned from the server are displayed on the user's screen. For emails, the search results include the subject, sender, recipient, and date and time, and for chats, a preview of the message is displayed.

[0052] 2. Users can click on individual search results to view more information. For example, clicking the Details button will open a modal window or new page with the full text of the email or chat history.

[0053] Specific examples

[0054] scenario:

[0055] 1. User A wants to investigate past response history regarding "Project Y" and accesses the system's search interface.

[0056] 2. User A enters "Project Y" as a keyword and sets the date range to the past 6 months.

[0057] 3. When the search button is pressed, the server searches the database for emails and chat history related to "Project Y," formats it, and returns it to User A's device.

[0058] 4. User A reviews the list of search results, clicks on the subject of the relevant email to read the full text and extract the key points.

[0059] In this way, the proposed system quickly and efficiently integrates distributed communication histories, allowing users to easily obtain the information they need.

[0060] The processing flow will be explained below.

[0061] Step 1: Get the data

[0062] server:

[0063] The server is configured to periodically retrieve new messages and historical data using the APIs or protocols of various communication tools (e.g., IMAP, POP3).

[0064] To retrieve new mail, the IMAP protocol is used to retrieve new mail from each user's inbox.

[0065] Call the chat tool's public API to get new messages from that tool.

[0066] Step 2: Transform the data

[0067] server:

[0068] The acquired data is temporarily saved and converted into a unified format (e.g., JSON format).

[0069] In the case of email data, information such as the subject, body, sender / receiver, date and time is extracted.

[0070] For chat data, message content, sender, recipient, and timestamp are extracted.

[0071] Step 3: Save your data

[0072] server:

[0073] The converted data is checked for duplicates and saved in the database.

[0074] Save new data to the database using INSERT operations and, if necessary, update operations.

[0075] Index the database to speed up later searches.

[0076] Step 4: Enter search criteria

[0077] User:

[0078] Users access a search interface and enter criteria such as search keywords, date ranges, and specific senders and recipients.

[0079] The interface allows you to enter search keywords into the text box and set a date range using the calendar widget.

[0080] Click the search button.

[0081] Step 5: Generating a search query

[0082] server:

[0083] It receives search requests from users and generates SQL queries based on the conditions.

[0084] For example, dynamically generate a SQL statement specifying keywords and a date range related to "Project X."

[0085] The generated query is sent to the database.

[0086] Step 6: Database Search

[0087] server:

[0088] Run the generated SQL queries against the database to find relevant historical data.

[0089] Search results are sorted chronologically and by relevance.

[0090] Step 7: Formatting the search results

[0091] server:

[0092] Format the search results and create a response to return to the user.

[0093] Generates JSON and HTML responses and includes additional information for each search result (e.g., email previews, chat message excerpts).

[0094] Step 8: Viewing search results

[0095] Device:

[0096] The search results returned from the server are displayed on the user's screen.

[0097] Results can be sorted chronologically or by relevance, and for emails, the subject, sender, sender, and date and time are displayed.

[0098] Step 9: View details and take action

[0099] User:

[0100] Users click on search result items to see more details.

[0101] View a specific email or chat history in a modal window or new page, with other related messages displayed as links.

[0102] Example 1

[0103] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0104] Conventional information retrieval systems face the problem of difficulty in centrally managing historical information from various information transmission tools, making it difficult to quickly search for and display specific information. Furthermore, performance often declines when processing search requests from multiple users in parallel. Another issue is the lack of a means to effectively sort and display search results.

[0105] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0106] In this invention, the server includes means for acquiring history information from various information transmission tools, means for converting the history information into a unified format, means for storing the information converted into the unified format in a data warehouse, means for searching the data warehouse for corresponding history information based on search conditions specified by a user, means for visualizing the searched history information to the user, means for periodically acquiring history information, means for checking for duplication and storing the information, means for accessing a dedicated search interface, means for inputting search keywords, date ranges, and specific senders and receivers, means for generating a query language based on the conditions, and means for sorting the acquired data in chronological order or by relevance. This makes it possible to efficiently collect, manage, and search history information from distributed information transmission tools and quickly respond to search requests from multiple users.

[0107] "Information communication tools" refers to communication methods such as email, chat, and messaging applications.

[0108] "History information" refers to messages and data sent and received through various information transmission tools, as well as related metadata.

[0109] "Unified format" refers to a format that converts data obtained from different information delivery tools into a consistent format.

[0110] "Data warehouse" refers to a database system designed to store and manage acquired historical information and enable efficient search and extraction.

[0111] "Search conditions" refer to conditions such as keywords, date ranges, senders, and recipients that a user specifies when searching for specific information.

[0112] "Duplicate check" refers to a confirmation process to prevent the same history information from being saved twice.

[0113] "Search interface" refers to a user interface that allows a user to input search criteria and view search results.

[0114] "Query language" refers to a query language (e.g., SQL) used to retrieve data from a database under specific conditions.

[0115] "Chronological order" refers to the order in which data is arranged in the order in which it occurred.

[0116] "Sort by relevance" refers to the order in which data is arranged in descending order of relevance to the search criteria.

[0117] This invention provides a specific method for implementing a system that acquires historical information from various information transmission tools, converts it into a unified format, stores it in a data warehouse, searches for corresponding historical information based on search conditions specified by the user, and visualizes it for the user.

[0118] First, the server periodically collects historical information from multiple information transmission tools. It retrieves emails from the mail server using the IMAP or POP3 protocol, and retrieves messages from the chat tool via API. It uses a cron job to check for new data every hour. The collected data is temporarily stored in local storage.

[0119] The server then runs a parser to convert the collected history information into a unified format. For example, a Python script is used to extract information such as the subject, body, sender, recipient, and date and time from emails and convert them into JSON objects. Similarly, chat messages are converted into a unified format, with information such as the message content, sender, recipient, and timestamp.

[0120] The server checks for duplicates of the converted data before storing it in a data warehouse (e.g., MySQL (registered trademark), PostgreSQL). Message IDs and timestamps are used to check for duplicates. Data stored in the data warehouse is designed to be efficiently searchable and extractable.

[0121] Users access a dedicated search interface and enter search criteria (e.g., search keywords, date ranges, specific senders / receivers). The search interface is provided using text boxes, drop-down menus, and calendar widgets. When a search button is pressed, the server receives the search request and generates an SQL query based on the received criteria.

[0122] For example, if the conditions "Project Y" and "Last 6 months" are specified, the server generates an SQL query such as "SELECT FROM communication_logs WHERE body LIKE '%ProjectY%' AND timestamp BETWEEN® 'Start Date' AND 'End Date'". The server then executes the query on the database and retrieves the search results. The retrieved results are sorted chronologically and by relevance.

[0123] The device displays the search results returned by the server on the user's screen. For emails, the subject, sender, recipient, and date and time are displayed, and for chats, a preview of the message is displayed. The user can click on individual search results to view more information.

[0124] Specific examples

[0125] scenario:

[0126] 1. User A wants to investigate past response history regarding "Project Y" and accesses the system's search interface.

[0127] 2. User A enters "Project Y" as a keyword and sets the date range to the past 6 months.

[0128] 3. When the search button is pressed, the server searches the data warehouse for emails and chat history related to "Project Y," formats the data, and returns it to User A's device.

[0129] 4. User A reviews the list of search results, clicks on the subject of the relevant email to read the full text and extract the key points.

[0130] Example prompts for generative AI models

[0131] "Please explain the algorithm for a system that converts historical information obtained from various information transmission tools into a unified format and stores it in a data warehouse. This includes the ability for users to search for specific keywords or date ranges."

[0132] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0133] Step 1:

[0134] The server obtains history information from the information transmission tool. For example, it obtains messages from the mail server using the IMAP or POP3 protocol, and from the chat tool via an API. Input data includes authentication information for the mail server and the API key for the chat tool. Based on this, new emails and messages are temporarily saved in local storage.

[0135] Step 2:

[0136] The server reads the history information from the temporary storage folder and converts it into a unified format. The input data is history information files in JSON and CSV format. Specifically, a Python script is executed to extract the email subject, body, sender / receiver, date and time, etc., and convert them into a unified JSON object. The output data is a unified JSON object containing the subject, body, sender, recipient, and timestamp information.

[0137] Step 3:

[0138] The server checks for duplicates in the converted data before storing it in the data warehouse. The input data is a unified JSON object. An SQL statement such as "INSERT INTO communication_logs (subject, body, sender, receiver, timestamp) VALUES (...) ON DUPLICATE KEY UPDATE ..." is executed against the database. The output is accurate historical information stored in the data warehouse.

[0139] Step 4:

[0140] Users access a dedicated search interface and enter search criteria. The input data can include search keywords, a date range, and specific senders and recipients. Based on this, the search interface receives the entered criteria and displays them on the screen. The output is the received search criteria.

[0141] Step 5:

[0142] The server receives a search request from the user and generates an SQL query based on the conditions. For example, if "Project Y" and "Last 6 months" are specified, it generates an SQL query such as "SELECT FROM communication_logs WHERE body LIKE '%ProjectY%' AND timestamp BETWEEN 'Start date' AND 'End date'". The input data is the search conditions, and the output data is the generated SQL query.

[0143] Step 6:

[0144] The server executes the generated SQL query and retrieves the search results from the data warehouse. This involves executing the SQL query against the database and retrieving the relevant data. The input data is the SQL query, and the output data is the dataset of the search results.

[0145] Step 7:

[0146] The server sorts and formats the search results in chronological order or relevance order. For example, it sorts the list of search results by timestamp. The input data is the dataset of search results, and the output data is the sorted dataset.

[0147] Step 8:

[0148] The device displays the search results returned by the server on the user's screen: for emails, the subject, sender, recipient, and date and time are displayed, and for chats, a preview of the message is displayed. The input data is a sorted dataset, and the output is the search results displayed in the user's browser.

[0149] Step 9:

[0150] The user selects the information they need from the displayed list of search results and checks the details. For example, they click on the subject of a specific email to read the full text and extract important points. The input data is the displayed search results, and the output data is the email content and chat history displayed in the detailed view.

[0151] (Application example 1)

[0152] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0153] Currently, security service inquiries and trouble report history management is distributed across multiple communication tools (email, chat, etc.), and there is an inadequate system for integrated management and search. As a result, information sharing between end users and support teams is delayed, making it difficult to respond quickly. Furthermore, when searching for related data, users often cannot specify appropriate search criteria, which often hinders efficient information retrieval. These issues need to be resolved.

[0154] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0155] In this invention, the server includes means for acquiring history data from various communication tools, means for converting the history data into a unified format, means for storing the data converted into the unified format in a database, means for searching the database for corresponding history data based on search criteria specified by a user, means for visualizing the searched history data to the user, means for acquiring history data using data collection software and interface software, converting the data into a unified format, and storing the data, means for performing search and visualization through an application compatible with the user's smart device, and means for recommending search criteria and automatically generating prompt sentences using a generative AI model. This allows for centralized management of distributed communication histories between end users and support teams, enabling rapid and appropriate information search and sharing.

[0156] "Various communication tools" is a general term for communication methods using different media, such as email, chat applications, and messaging platforms.

[0157] "Historical Data" means records of messages, emails, and other communications exchanged through various communication tools.

[0158] A "uniform format" is a standardized data format for converting different types of historical data into a consistent format.

[0159] "Database" means a centralized data management system for efficiently storing, searching, and managing historical data that has been converted into a unified format.

[0160] "Search criteria" are criteria such as keywords, date ranges, senders, and recipients that a user specifies to identify and search for specific historical data.

[0161] "Visualization" is the process of displaying retrieved historical data in a way that is easy for users to understand.

[0162] "Data collection software" is a program for automatically collecting historical data from various communication tools.

[0163] "Interface software" refers to the operating screens and applications that allow users to interact with the data collection software and database.

[0164] "Smart devices" are highly functional portable devices such as smartphones, tablets, smart glasses, and head-mounted displays.

[0165] A "generative AI model" is a computational model that uses artificial intelligence techniques to automatically perform specific tasks.

[0166] "Search criterion recommendation" is a function in which the generative AI model automatically suggests appropriate search criteria so that users can specify them.

[0167] "Automatic prompt generation" is the process of using a generative AI model to automatically generate commands that users use when searching or performing operations.

[0168] The present invention provides a system that uses a user's smart device to centrally manage, search, and visualize historical data of inquiries and trouble reports for security services. Specific embodiments of this system will be described below.

[0169] Specific methods for collecting data

[0170] server:

[0171] The server periodically retrieves historical data from various communication tools. For example, it retrieves emails from a mail server using the IMAP or POP3 protocol, and retrieves messages from chat tools using an API. A parser is used to convert the retrieved data into a unified format. In the case of email data, information such as the subject, body, sender / receiver, and date / time is retrieved, and in the case of chat data, information on the message content, sender, recipient, and timestamp is retrieved. The converted data is stored in a database. This database checks for duplicates and manages the data in a unified format.

[0172] Specific methods for data search

[0173] User:

[0174] Users access a dedicated search interface and enter search criteria such as search keywords, date ranges, and specific senders and recipients. The interface is provided through a smart device application. When the user presses the search button, the server receives the search request and generates an SQL query based on the criteria. The generated SQL query is used to search corresponding historical data from the database and retrieve search results. The search results are then sorted in chronological order or by relevance.

[0175] Specific display and operation methods

[0176] Device:

[0177] The application installed on the user's smart device displays the search results returned from the server on the user's screen. For emails, the search results display the subject, sender, recipient, and date and time, while for chats, a preview of the message is displayed. The user can click on individual items in the search results to view more information. Pressing the details button displays the full text of the relevant email or chat history.

[0178] Utilizing generative AI models and prompts

[0179] Generative AI models:

[0180] The server uses a generative AI model to provide a function that recommends search terms entered by the user. For example, if a user enters an ambiguous keyword, the server automatically suggests specific search terms related to that keyword. Furthermore, the server also automatically generates prompt sentences to help users perform searches efficiently.

[0181] Examples:

[0182] User A wants to investigate past correspondence history regarding "Project Y" and accesses the system's search interface. User A enters "Project Y" as a keyword and sets the date range to the past six months. When he presses the search button, the server searches the database for emails and chat history related to "Project Y," formats it, and returns it to User A's device. User A checks the list of search results displayed, clicks on the subject of a related email to read the full text, and extracts important points.

[0183] Example prompt sentence:

[0184] A user wants to search for past security-related incident history using specific keywords. When the user enters "Keyword: security incident" and "Date range: January 1, 2023 to December 31, 2023", generate Python code to retrieve the above historical data and display relevant results.

[0185] In this way, the proposed system quickly and efficiently integrates distributed communication histories, allowing users to easily obtain the information they need.

[0186] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0187] Step 1:

[0188] The server acquires historical data from various communication tools (e.g., mail servers, chat APIs). Specifically, it acquires email data using the IMAP or POP3 protocol, and message data using the chat tool's API. The input is raw data from each communication tool, and the output is the acquired raw data.

[0189] Step 2:

[0190] The server converts the acquired raw data into a unified format. For example, email data is broken down into items such as subject, body, sender, sender, and date and time, while chat data is broken down into message content, sender, receiver, and timestamp information. The input is raw data, and the output is data converted into a unified format.

[0191] Step 3:

[0192] The server saves the data converted into a unified format in the database. At the same time, it checks for duplicates and removes items that overlap with existing data. The input is unified format data, and the output is the data saved in the database.

[0193] Step 4:

[0194] A user accesses a dedicated search interface installed on a smart device and inputs search criteria such as search keywords, date range, sender / receiver, etc. The input is the search criteria specified by the user, and the output is a search request sent to the search interface.

[0195] Step 5:

[0196] The server receives a search request from a user, generates an SQL query based on the conditions, and searches the corresponding historical data from the database. The input is the search request from the user, and the output is the historical data in the database that matches the search conditions.

[0197] Step 6:

[0198] The server sorts the search results in chronological order or by relevance. The input is the searched history data, and the output is the sorted search results.

[0199] Step 7:

[0200] The server uses a generative AI model to automatically generate search query recommendations and prompts for users. The input is the initial search query and data in the database, and the output is the recommended search query and prompt.

[0201] Step 8:

[0202] The terminal displays the search results returned from the server. For emails, the search results show the subject, sender, recipient, and date and time, and for chats, a preview of the message is displayed. The input is the search results from the server, and the output is the search results in list format that is displayed on the user's screen.

[0203] Step 9:

[0204] Users can click on individual items from the displayed search results to check detailed information. Pressing the details button displays the full text of the relevant email or chat history. The input is the user's click, and the output is a popup or separate screen displaying detailed information.

[0205] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0206] This invention combines a system that acquires history data from various communication tools, converts it into a unified format, stores it in a database, searches for corresponding history data based on search conditions specified by the user, and visualizes it for the user with an emotion engine that recognizes the user's emotions. This system can adjust search results and change presentation methods based on the user's emotions.

[0207] Data collection

[0208] server:

[0209] 1. The server obtains historical data from various communication tools (e.g., mail servers, chat servers). Examples include obtaining new emails from a mail server using the IMAP or POP3 protocol, or obtaining messages using a chat tool's API.

[0210] 2. To convert the acquired data into a unified format, the subject, body, sender, sender, and date / time are extracted from email data, and the message content, sender, receiver, and timestamp information are extracted from chat data.

[0211] 3. Save the converted data in the database and check for duplicates.

[0212] Data Search

[0213] User:

[0214] 1. The user accesses a dedicated search interface and enters search criteria such as search keywords, a date range, or specific senders / receivers. The input interface uses text boxes, drop-down menus, and calendar widgets.

[0215] 2. Set the search conditions and press the search button.

[0216] server:

[0217] 1. The server receives a search request from a user and generates a SQL query, for example, specifying the keyword "Project X" and a specific date range.

[0218] 2. Run the generated SQL query against the database to retrieve search results, sorted chronologically and / or by relevance.

[0219] Emotion engine processing

[0220] server:

[0221] 1. Before formatting the search results, we use an emotion engine to recognize the user's emotions by analyzing the user's speech and input text when entering search criteria.

[0222] 2. Tailor search results based on perceived emotions, for example, prioritizing more important information if a user is feeling stressed.

[0223] 3. The presentation of search results will also change based on emotion, for example, results will be displayed in vibrant colors for positive emotions and muted colors for negative emotions.

[0224] Display and Operation

[0225] Device:

[0226] 1. The search results returned from the server are displayed on the user's screen. For emails, the results show the subject, sender, recipient, and date and time, and for chats, a preview of the message is displayed.

[0227] 2. Users can click on search results to learn more, displaying a specific email or chat history in a modal window or new page, with other related messages available as links.

[0228] Specific examples

[0229] scenario:

[0230] 1. User A accesses the system's search interface to investigate past response history regarding "Project Y."

[0231] 2. User A enters "Project Y" as a keyword and sets the date range to the past six months. The emotion engine recognizes from User A's voice and text that he or she is feeling stressed.

[0232] 3. When the search button is pressed, the server searches the database for emails and chat history related to "Project Y," prioritizes the most important information based on the stress level recognized by the emotion engine, and returns it to User A's device.

[0233] 4. User A reviews the list of search results, clicks on the subject of the relevant email to read the full text and extract the key points.

[0234] In this way, the proposed system takes into account the user's emotions, quickly and efficiently integrates distributed communication histories, and allows users to easily obtain the information they need.

[0235] The processing flow will be explained below.

[0236] Step 1: Get the data

[0237] server:

[0238] The server is configured to periodically retrieve new messages and historical data using the APIs or protocols of various communication tools (e.g., IMAP, POP3).

[0239] To retrieve new mail, the IMAP protocol is used to retrieve new mail from each user's inbox.

[0240] Call the chat tool's public API to get new messages from that tool.

[0241] Step 2: Transform the data

[0242] server:

[0243] The acquired data is temporarily saved and converted into a unified format (e.g., JSON format).

[0244] In the case of email data, information such as the subject, body, sender / receiver, date and time is extracted.

[0245] For chat data, message content, sender, recipient, and timestamp are extracted.

[0246] Step 3: Save your data

[0247] server:

[0248] The converted data is checked for duplicates and saved in the database.

[0249] Save new data to the database using INSERT operations and, if necessary, update operations.

[0250] Index the database to speed up later searches.

[0251] Step 4: Enter search criteria

[0252] User:

[0253] Users access a search interface and enter criteria such as search keywords, date ranges, and specific senders and recipients.

[0254] The interface allows you to enter search keywords into the text box and set a date range using the calendar widget.

[0255] Click the search button.

[0256] Step 5: Recognize emotions

[0257] server:

[0258] The emotion engine recognizes the user's emotions through text analysis of the user's input (keywords and comments) or, in the case of voice input, through voice analysis.

[0259] Data about the recognized emotions is temporarily stored for use in adjusting search queries and result display.

[0260] Step 6: Generating a search query

[0261] server:

[0262] It receives a search request from the user and generates an SQL query taking into account the results of the emotion engine.

[0263] For example, dynamically generate SQL statements specifying keywords related to "Project X" and a specific date range, and prioritize the importance of the results if the user's sentiment is negative.

[0264] Step 7: Database Search

[0265] server:

[0266] Run the generated SQL queries against the database to find relevant historical data.

[0267] Search results are sorted chronologically and by relevance based on user sentiment.

[0268] Step 8: Formatting the search results

[0269] server:

[0270] Format the search results and create a response to return to the user.

[0271] Generates JSON and HTML responses and includes additional information for each search result (e.g., email previews, chat message excerpts).

[0272] Based on the results of the emotion engine, the way search results are displayed (color tone, layout, etc.) is adjusted.

[0273] Step 9: Viewing search results

[0274] Device:

[0275] The search results returned from the server are displayed on the user's screen.

[0276] Results can be sorted chronologically or by relevance, and for emails, the subject, sender, sender, and date and time are displayed.

[0277] Step 10: View details and operate

[0278] User:

[0279] Users click on search result items to see more details.

[0280] View a specific email or chat history in a modal window or new page, with other related messages displayed as links.

[0281] Example 2

[0282] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0283] Conventional communication history search systems make it difficult for users to efficiently retrieve the information they need from the vast number of emails and chat histories. Furthermore, they are unable to prioritize information display based on the user's emotions or change the display format, so there is a need for improved user experience. Furthermore, processing search requests from multiple users simultaneously places a heavy load on the system, resulting in a decrease in system performance.

[0284] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0285] In this invention, the server includes means for acquiring communication history data from various information communication devices, means for converting the communication history data into a standard format, means for storing the data converted into the standard format in a storage device, means for searching the storage device for corresponding communication history data based on search criteria specified by a user, means for displaying the searched communication history data to the user, means for recognizing the user's emotion, means for adjusting the search results based on the recognized emotion, and means for changing the display format of the search results based on the recognized emotion. This enables prioritized display of information and change of the display format according to the user's emotion, allowing necessary information to be obtained quickly and efficiently. System performance is also improved when simultaneously processing search requests from multiple users.

[0286] An "information communication device" is a device that provides communication history data, such as an email server or chat server.

[0287] "Communication history data" refers to data that includes information such as the content of emails and chat messages, senders and receivers, and dates and times.

[0288] The "standard format" is a data format for converting communication history data in different formats into a unified format.

[0289] The "storage device" is a storage medium such as a database for storing communication history data.

[0290] "Search criteria" are search criteria such as keywords, date ranges, senders and recipients specified by the user.

[0291] The "display means" is an interface for visually presenting search results to the user.

[0292] "Emotion recognition" is a technology that analyzes and recognizes the emotions expressed by users when they enter search criteria.

[0293] "Search result tailoring" refers to changing the priorities of search results based on perceived user sentiment.

[0294] "Changing the display format" means changing the presentation method of search results according to the user's emotions.

[0295] This invention combines a system that acquires history data from various information and communication devices, converts it into a standard format, stores it in a storage device, searches for corresponding history data based on search conditions specified by the user, and visualizes it for the user with an emotion engine that recognizes the user's emotions. This system can adjust search results and change the presentation method based on the user's emotions.

[0296] The server first obtains communication history data from various information and communication devices (e.g., mail servers, chat servers). Specifically, it obtains new emails from the mail server using the IMAP or POP3 protocol, and then obtains messages using the chat tool's API. Specific software libraries called IMAPClient and Requests are used.

[0297] To convert the retrieved data into a standard format, extract the subject, body, sender, recipient, and date and time from email data, and extract the message content, sender, recipient, and timestamp information from chat data. To extract the subject from email data, use email.message_from_string(raw_email)['Subject'], and to extract the message content from chat data, use json.loads(response.text)['messages'].

[0298] Next, the converted data is saved to a storage device (database) and checked for duplicates. To save to the database, an SQL query (e.g., INSERT INTO communications (subject, body, sender, recipient, date) VALUES ...) is executed. A duplicate check is performed using an SQL query (e.g., SELECT COUNT() FROM communications WHERE ...).

[0299] Users access a specialized search interface and enter search criteria, such as search keywords, date ranges, specific senders and recipients, using text boxes, drop-down menus, and calendar widgets. When the user presses a search button, the search criteria are sent to the server.

[0300] Next, the server receives a search request from the user and generates an SQL query based on the search criteria, for example, sql_query = f"SELECT FROM communications WHERE subject LIKE '%{keyword}%' AND date BETWEEN '{start_date}' AND '{end_date}'". It executes this SQL query, retrieves the results, and sorts them by chronological order or relevance.

[0301] Furthermore, the server uses an emotion engine to recognize the user's emotions before formatting the search results. To recognize the user's emotions, it uses an emotion recognition API such as emotion = analyze_emotion(user_input). Based on the recognized emotion, it adjusts the search results. For example, if stress is recognized, it reorders the search results using the method results = prioritize_important_information(results). The presentation method of the search results is also changed accordingly.

[0302] Search results are displayed on the user's device. For emails, the results show the subject, sender, recipient, and date and time, and for chats, a preview of the message is displayed. Users can click on a result to view more information. The history of a specific email or chat can be displayed in a modal window or on a new page, with other related messages displayed as links.

[0303] Specific examples

[0304] 1. User A accesses the system's search interface to investigate past response history regarding "Project Y."

[0305] 2. User A enters "Project Y" as a keyword and sets the date range to the past six months. The emotion engine recognizes from User A's voice and text that he or she is feeling stressed.

[0306] 3. When the search button is pressed, the server searches the database for emails and chat history related to "Project Y," prioritizes the most important information based on the stress level recognized by the emotion engine, and returns it to User A's device.

[0307] 4. User A reviews the list of search results, clicks on the subject of the relevant email to read the full text and extract the key points.

[0308] Prompt Sentence Examples

[0309] I want to investigate the past history of projects related to Project Y. The emotion engine should recognize the user's emotions and prioritize the most important information. Then, display the search results based on color and presentation style.

[0310] In this way, the proposed system takes into account the user's emotions, quickly and efficiently integrates distributed communication histories, and allows users to easily obtain the information they need.

[0311] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0312] Step 1:

[0313] server:

[0314] Input: Connection information and authentication information obtained from various information and communication devices.

[0315] Operation: The server connects to various information and communication devices (e.g., mail servers, chat servers) and acquires communication history data. Specifically, it acquires new emails from the mail server using the IMAP or POP3 protocol, and acquires messages using the chat tool's API.

[0316] Output: Acquired communication history data (e.g., content of emails and chat messages, sender / receiver, date and time, etc.).

[0317] Step 2:

[0318] server:

[0319] Input: Acquired communication history data (e.g., raw email data, chat message data).

[0320] Operation: The server converts the acquired communication history data into a standard format. From email data, it extracts the subject, body, sender, recipient, and date and time, and from chat data, it extracts the message content, sender, recipient, and timestamp information. Specifically, to extract the subject from email data, it uses email.message_from_string(raw_email)['Subject'], and to extract the message content from chat data, it uses json.loads(response.text)['messages'].

[0321] Output: Communication history data converted into a standard format.

[0322] Step 3:

[0323] server:

[0324] Input: Communication history data converted into a standard format.

[0325] Operation: The server converts the communication history data into a standard format, stores it in a storage device (database), and checks for duplicates. Specifically, to store it in the database, it executes the SQL query INSERT INTO communications (subject, body, sender, recipient, date) VALUES ..., and to check for duplicates, it uses the SQL query SELECT COUNT() FROM communications WHERE ....

[0326] Output: Saved communication history data and whether there are any duplicates.

[0327] Step 4:

[0328] User:

[0329] Input: Access the search interface, search criteria such as search keywords, date range, or specific senders / receivers.

[0330] How it works: Users access a dedicated search interface and enter search criteria using text boxes, drop-down menus, and calendar widgets.

[0331] Output: The search criteria that was set.

[0332] Step 5:

[0333] server:

[0334] Input: The search criteria submitted by the user.

[0335] How it works: The server receives a search request from a user and generates an SQL query based on the search criteria. Specifically, it generates an SQL query in the format sql_query = f"SELECT FROM communications WHERE subject LIKE '%{keyword}%' AND date BETWEEN '{start_date}' AND '{end_date}'".

[0336] Output: The generated SQL query.

[0337] Step 6:

[0338] server:

[0339] Input: The generated SQL query.

[0340] How it works: The server executes an SQL query and retrieves search results. The server then executes an SQL query to sort the results in chronological order or relevance order, such as ORDER BY date DESC.

[0341] Output: Sorted search results.

[0342] Step 7:

[0343] server:

[0344] Input: Sorted search results.

[0345] Operation: The server uses the emotion engine to recognize the user's emotions. Specifically, it analyzes the text and voice input of the user's search criteria and calls the emotion recognition API, emotion = analyze_emotion(user_input).

[0346] Output: Perceived user sentiment.

[0347] Step 8:

[0348] server:

[0349] Input: Perceived user sentiment, sorted search results.

[0350] How it works: The server tailors search results based on the perceived emotion. For example, if stress is perceived, it reorders the search results to prioritize more important information using the method results = prioritize_important_information(results).

[0351] Output: Refined search results.

[0352] Step 9:

[0353] server:

[0354] Input: Perceived user sentiment, adjusted search results.

[0355] How it works: The server changes the presentation of search results based on the recognized emotion, for example, presenting vibrant colors for positive emotions and calm colors for negative emotions, using theme-changing methods like if emotion == 'positive': set_vibrant_theme() else: set_calm_theme().

[0356] Output: The format in which the final search results are displayed.

[0357] Step 10:

[0358] Device:

[0359] Input: The format in which the final search results will be displayed.

[0360] What it does: Search results returned by the server are displayed on the user's screen. For emails, the search results show the subject, sender / receiver, and date and time, and for chats, a preview of the message is displayed.

[0361] Output: Search results displayed on the user's screen.

[0362] Step 11:

[0363] User:

[0364] Input: The search result item the user selects on the screen.

[0365] What it does: Users click on a search result item to learn more, or view a specific email or chat history in a modal window or new page, with other related messages available as links.

[0366] Output: A modal window or new page with more information.

[0367] (Application example 2)

[0368] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0369] Conventional food delivery systems only provide information based on order history, support interaction history, and review comments without considering the user's emotions. As a result, when a user is stressed or in a particular emotional state, appropriate information is not provided, resulting in a decrease in user satisfaction.

[0370] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0371] In this invention, the server includes means for acquiring history data from various communication means, means for converting the history data into a unified format, means for storing the data converted into the unified format in a data storage means, means for searching for corresponding history data from the data storage means based on search conditions specified by a user, means for visualizing the searched history data to the user, an emotion recognition engine for analyzing the user's emotions, means for adjusting search results based on the emotions, and means for changing the presentation format based on the emotions, thereby making it possible to provide appropriate information according to the user's emotions.

[0372] "Various communication means" refers to means for exchanging information via multiple different communication protocols and devices.

[0373] "History data" refers to information about past interactions with a user, and primarily includes order history, support interaction history, review comments, and the like.

[0374] A "uniform format" is a standardized data format for consistently handling data acquired in different formats.

[0375] "Data storage means" refers to a means for saving acquired historical data, and typically includes databases and cloud storage.

[0376] "Search conditions" are key information that a user specifies when retrieving specific data, and include keywords, date ranges, specific senders and receivers, and the like.

[0377] The "visualization means" is a means for visually displaying search results to the user, such as a graphical user interface.

[0378] An "emotion recognition engine" is an engine that has the function of analyzing a user's emotions, and is a technology that analyzes emotions from a user's speech or text input.

[0379] The "means for adjusting search results based on emotions" refers to a means for changing the content and display order of search results according to the user's emotions analyzed by the emotion recognition engine.

[0380] The "means for changing the presentation method based on emotions" is a means for adjusting the display method of search results and the UI theme based on the user's emotions.

[0381] The embodiments for carrying out the present invention are as follows.

[0382] Data collection and storage

[0383] The server has a means of acquiring historical data from various communication methods. Specifically, data such as email, chat messages, and order history is acquired using multiple APIs and communication protocols (e.g., IMAP, POP3, HTTP). The acquired historical data is converted into a unified format. For example, the subject, body, sender, sender, and date / time are extracted from email, and the message content, sender, receiver, and timestamp information are extracted from chat messages. The converted data is then saved in an SQLite database, which serves as the data storage method. A check for duplicate data is also performed when saving the data.

[0384] Data Search

[0385] A user accesses a dedicated search interface and enters search criteria. For example, in a food delivery service application, a user can specify keywords and a date range for order history. The server receives the user's search request and generates an SQL query. For example, an SQL statement specifying the keyword "order" and a specific date range is generated. The generated SQL query is executed against the database to retrieve the corresponding history data.

[0386] Sentiment Analysis Engine

[0387] When a user enters search criteria, the emotion recognition engine analyzes the user's emotions. This includes the text the user enters and speech using speech recognition technology. This determines the emotional state as positive, negative, or neutral. The emotion recognition engine uses TextBlob and other natural language processing libraries.

[0388] Tailoring and displaying search results

[0389] The server adjusts search results based on the emotions analyzed by the emotion recognition engine. For example, if the user is feeling stressed, it prioritizes displaying information that is of high importance. It also changes the presentation style based on the emotion, applying a brightly colored design for positive emotions and a more subdued design for negative emotions. The adjusted search results are visualized on the user's device. The results may include, for example, order history details or support interaction history, displayed in chronological order or by relevance.

[0390] Specific examples

[0391] If a user types "my order is late," the server uses an emotion recognition engine to analyze negative emotions. It then adjusts search results, prioritizing important information and visualizing the results in calming colors. Users can then respond quickly based on the displayed results, reducing stress.

[0392] Prompt Sentence Examples

[0393] Use the Food Delivery Emotional Assistant to investigate my pizza order history. I'm stressed because my order is late.

[0394] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0395] Step 1:

[0396] The server acquires historical data from various communication methods. Specifically, it collects new orders, chat messages, and review comments via APIs such as mail servers and chat servers. It receives response data from the API as input and generates a list of the acquired raw data as output.

[0397] Step 2:

[0398] The server converts the acquired historical data into a unified format. Specifically, it extracts the subject, body, sender, recipient, and date and time from email data, and the message content, sender, recipient, and timestamp information from chat messages. It receives a list of raw data as input and generates a list of standardized data as output.

[0399] Step 3:

[0400] The server saves the converted data in a database, which is a data storage means. Specifically, it inserts the data into an SQLite database and also checks for duplicates. It receives a list of standardized data as input and generates the number of records of the data stored in the database as output.

[0401] Step 4:

[0402] A user accesses a dedicated search interface and enters search criteria, such as search keywords and date ranges. The system takes the search criteria specified by the user as input and generates a query based on the search criteria as output.

[0403] Step 5:

[0404] The server generates an SQL query based on the search criteria it receives. Specifically, it creates an SQL statement that includes search keywords and date ranges. It takes the search criteria specified by the user as input and generates an executable SQL statement as output.

[0405] Step 6:

[0406] The server executes the generated SQL query against the database to search for the corresponding historical data. Specifically, it retrieves the relevant records from the database and sorts them in chronological order or by relevance. It receives the SQL statement as input and generates a list of search results as output.

[0407] Step 7:

[0408] The server uses an emotion recognition engine to analyze the user's emotions. Specifically, it analyzes the text and voice input by the user and determines whether the emotion is positive, negative, or neutral. It receives the user's text and voice data as input and generates the type of emotion as output.

[0409] Step 8:

[0410] The server adjusts search results based on the user's emotions. Specifically, it sorts the search results to prioritize the most important information. As input, it receives the emotion type and a list of search results, and as output, it generates a list of adjusted search results.

[0411] Step 9:

[0412] The server displays the adjusted search results in an appropriate presentation style, specifically, applying a vibrant design for positive emotions and a subdued design for negative emotions. The server receives the adjusted search results list as input and generates a search result screen to be displayed on the user's device as output.

[0413] Step 10:

[0414] The user can interact with the displayed search results and view more information. For example, they can click on a specific order history or support interaction history to view its contents. The system takes the displayed search results as input and displays a modal window or a new page with more information as output.

[0415] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0416] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0417] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0418] [Second embodiment]

[0419] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0420] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0421] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0422] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0423] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0424] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0425] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0426] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0427] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0428] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0429] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0430] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0431] This invention provides a specific method for implementing a system that acquires historical data from various communication tools, converts it into a unified format, stores it in a database, searches for corresponding historical data based on search conditions specified by the user, and visualizes it for the user.

[0432] Data collection

[0433] server:

[0434] 1. The server periodically retrieves historical data from each communication tool (e.g., mail server, chat server). This includes retrieving new emails from the mail server using the IMAP or POP3 protocol, and retrieving messages using the chat tool's API.

[0435] 2. A parser is used to convert the retrieved data into a unified format. For example, for email data, information such as the subject, body, sender, recipient, and date and time is retrieved, and for chat data, message content, sender, recipient, and timestamp information is retrieved.

[0436] 3. The converted data is saved in the database. This data is checked for duplicates and saved in the database in a unified format.

[0437] Data Search

[0438] User:

[0439] 1. The user accesses a dedicated search interface and enters search criteria such as search keywords, a date range, or specific senders / receivers. The interface is presented using text boxes, drop-down menus, and calendar widgets.

[0440] 2. Set the search conditions and press the search button.

[0441] server:

[0442] 1. The server receives a search request from the user and generates a SQL query based on the criteria. For example, if you specify the keyword "Project X" and a specific date range, it generates a SQL statement to search for historical data that matches those criteria.

[0443] 2. The generated SQL query is executed against the database to retrieve search results, which are then sorted by chronological order and relevance.

[0444] Display and Operation

[0445] Device:

[0446] 1. The search results returned from the server are displayed on the user's screen. For emails, the search results include the subject, sender, recipient, and date and time, and for chats, a preview of the message is displayed.

[0447] 2. Users can click on individual search results to view more information. For example, clicking the Details button will open a modal window or new page with the full text of the email or chat history.

[0448] Specific examples

[0449] scenario:

[0450] 1. User A wants to investigate past response history regarding "Project Y" and accesses the system's search interface.

[0451] 2. User A enters "Project Y" as a keyword and sets the date range to the past 6 months.

[0452] 3. When the search button is pressed, the server searches the database for emails and chat history related to "Project Y," formats it, and returns it to User A's device.

[0453] 4. User A reviews the list of search results, clicks on the subject of the relevant email to read the full text and extract the key points.

[0454] In this way, the proposed system quickly and efficiently integrates distributed communication histories, allowing users to easily obtain the information they need.

[0455] The processing flow will be explained below.

[0456] Step 1: Get the data

[0457] server:

[0458] The server is configured to periodically retrieve new messages and historical data using the APIs or protocols of various communication tools (e.g., IMAP, POP3).

[0459] To retrieve new mail, the IMAP protocol is used to retrieve new mail from each user's inbox.

[0460] Call the chat tool's public API to get new messages from that tool.

[0461] Step 2: Transform the data

[0462] server:

[0463] The acquired data is temporarily saved and converted into a unified format (e.g., JSON format).

[0464] In the case of email data, information such as the subject, body, sender / receiver, date and time is extracted.

[0465] For chat data, message content, sender, recipient, and timestamp are extracted.

[0466] Step 3: Save your data

[0467] server:

[0468] The converted data is checked for duplicates and saved in the database.

[0469] Save new data to the database using INSERT operations and, if necessary, update operations.

[0470] Index the database to speed up later searches.

[0471] Step 4: Enter search criteria

[0472] User:

[0473] Users access a search interface and enter criteria such as search keywords, date ranges, and specific senders and recipients.

[0474] The interface allows you to enter search keywords into the text box and set a date range using the calendar widget.

[0475] Click the search button.

[0476] Step 5: Generating a search query

[0477] server:

[0478] It receives search requests from users and generates SQL queries based on the conditions.

[0479] For example, dynamically generate a SQL statement specifying keywords and a date range related to "Project X."

[0480] The generated query is sent to the database.

[0481] Step 6: Database Search

[0482] server:

[0483] Run the generated SQL queries against the database to find relevant historical data.

[0484] Search results are sorted chronologically and by relevance.

[0485] Step 7: Formatting the search results

[0486] server:

[0487] Format the search results and create a response to return to the user.

[0488] Generates JSON and HTML responses and includes additional information for each search result (e.g., email previews, chat message excerpts).

[0489] Step 8: Viewing search results

[0490] Device:

[0491] The search results returned from the server are displayed on the user's screen.

[0492] Results can be sorted chronologically or by relevance, and for emails, the subject, sender, sender, and date and time are displayed.

[0493] Step 9: View details and take action

[0494] User:

[0495] Users click on search result items to see more details.

[0496] View a specific email or chat history in a modal window or new page, with other related messages displayed as links.

[0497] Example 1

[0498] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0499] Conventional information retrieval systems face the problem of difficulty in centrally managing historical information from various information transmission tools, making it difficult to quickly search for and display specific information. Furthermore, performance often declines when processing search requests from multiple users in parallel. Another issue is the lack of a means to effectively sort and display search results.

[0500] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0501] In this invention, the server includes means for acquiring history information from various information transmission tools, means for converting the history information into a unified format, means for storing the information converted into the unified format in a data warehouse, means for searching the data warehouse for corresponding history information based on search conditions specified by a user, means for visualizing the searched history information to the user, means for periodically acquiring history information, means for checking for duplication and storing the information, means for accessing a dedicated search interface, means for inputting search keywords, date ranges, and specific senders and receivers, means for generating a query language based on the conditions, and means for sorting the acquired data in chronological order or by relevance. This makes it possible to efficiently collect, manage, and search history information from distributed information transmission tools and quickly respond to search requests from multiple users.

[0502] "Information communication tools" refers to communication methods such as email, chat, and messaging applications.

[0503] "History information" refers to messages and data sent and received through various information transmission tools, as well as related metadata.

[0504] "Unified format" refers to a format that converts data obtained from different information delivery tools into a consistent format.

[0505] "Data warehouse" refers to a database system designed to store and manage acquired historical information and enable efficient search and extraction.

[0506] "Search conditions" refer to conditions such as keywords, date ranges, senders, and recipients that a user specifies when searching for specific information.

[0507] "Duplicate check" refers to a confirmation process to prevent the same history information from being saved twice.

[0508] "Search interface" refers to a user interface that allows a user to input search criteria and view search results.

[0509] "Query language" refers to a query language (e.g., SQL) used to retrieve data from a database under specific conditions.

[0510] "Chronological order" refers to the order in which data is arranged in the order in which it occurred.

[0511] "Sort by relevance" refers to the order in which data is arranged in descending order of relevance to the search criteria.

[0512] This invention provides a specific method for implementing a system that acquires historical information from various information transmission tools, converts it into a unified format, stores it in a data warehouse, searches for corresponding historical information based on search conditions specified by the user, and visualizes it for the user.

[0513] First, the server periodically collects historical information from multiple information transmission tools. It retrieves emails from the mail server using the IMAP or POP3 protocol, and retrieves messages from the chat tool via API. It uses a cron job to check for new data every hour. The collected data is temporarily stored in local storage.

[0514] The server then runs a parser to convert the collected history information into a unified format. For example, a Python script is used to extract information such as the subject, body, sender, recipient, and date and time from emails and convert them into JSON objects. Similarly, chat messages are converted into a unified format, with information such as the message content, sender, recipient, and timestamp.

[0515] The server checks for duplicates of the converted data before storing it in a data warehouse (e.g., MySQL, PostgreSQL). Message IDs and timestamps are used to check for duplicates. Data stored in the data warehouse is designed to be efficiently searchable and retrievalable.

[0516] Users access a dedicated search interface and enter search criteria (e.g., search keywords, date ranges, specific senders / receivers). The search interface is provided using text boxes, drop-down menus, and calendar widgets. When a search button is pressed, the server receives the search request and generates an SQL query based on the received criteria.

[0517] For example, if the conditions "Project Y" and "Last 6 months" are specified, the server generates an SQL query such as "SELECT FROM communication_logs WHERE body LIKE '%ProjectY%' AND timestamp BETWEEN 'Start Date' AND 'End Date'". The server then queries the database and retrieves the search results, which are sorted chronologically and by relevance.

[0518] The device displays the search results returned by the server on the user's screen. For emails, the subject, sender, recipient, and date and time are displayed, and for chats, a preview of the message is displayed. The user can click on individual search results to view more information.

[0519] Specific examples

[0520] scenario:

[0521] 1. User A wants to investigate past response history regarding "Project Y" and accesses the system's search interface.

[0522] 2. User A enters "Project Y" as a keyword and sets the date range to the past 6 months.

[0523] 3. When the search button is pressed, the server searches the data warehouse for emails and chat history related to "Project Y," formats the data, and returns it to User A's device.

[0524] 4. User A reviews the list of search results, clicks on the subject of the relevant email to read the full text and extract the key points.

[0525] Example prompts for generative AI models

[0526] "Please explain the algorithm for a system that converts historical information obtained from various information transmission tools into a unified format and stores it in a data warehouse. This includes the ability for users to search for specific keywords or date ranges."

[0527] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0528] Step 1:

[0529] The server obtains history information from the information transmission tool. For example, it obtains messages from the mail server using the IMAP or POP3 protocol, and from the chat tool via an API. Input data includes authentication information for the mail server and the API key for the chat tool. Based on this, new emails and messages are temporarily saved in local storage.

[0530] Step 2:

[0531] The server reads the history information from the temporary storage folder and converts it into a unified format. The input data is history information files in JSON and CSV format. Specifically, a Python script is executed to extract the email subject, body, sender / receiver, date and time, etc., and convert them into a unified JSON object. The output data is a unified JSON object containing the subject, body, sender, recipient, and timestamp information.

[0532] Step 3:

[0533] The server checks for duplicates in the converted data before storing it in the data warehouse. The input data is a unified JSON object. An SQL statement such as "INSERT INTO communication_logs (subject, body, sender, receiver, timestamp) VALUES (...) ON DUPLICATE KEY UPDATE ..." is executed against the database. The output is accurate historical information stored in the data warehouse.

[0534] Step 4:

[0535] Users access a dedicated search interface and enter search criteria. The input data can include search keywords, a date range, and specific senders and recipients. Based on this, the search interface receives the entered criteria and displays them on the screen. The output is the received search criteria.

[0536] Step 5:

[0537] The server receives a search request from the user and generates an SQL query based on the conditions. For example, if "Project Y" and "Last 6 months" are specified, it generates an SQL query such as "SELECT FROM communication_logs WHERE body LIKE '%ProjectY%' AND timestamp BETWEEN 'Start date' AND 'End date'". The input data is the search conditions, and the output data is the generated SQL query.

[0538] Step 6:

[0539] The server executes the generated SQL query and retrieves the search results from the data warehouse. This involves executing the SQL query against the database and retrieving the relevant data. The input data is the SQL query, and the output data is the dataset of the search results.

[0540] Step 7:

[0541] The server sorts and formats the search results in chronological order or relevance order. For example, it sorts the list of search results by timestamp. The input data is the dataset of search results, and the output data is the sorted dataset.

[0542] Step 8:

[0543] The device displays the search results returned by the server on the user's screen: for emails, the subject, sender, recipient, and date and time are displayed, and for chats, a preview of the message is displayed. The input data is a sorted dataset, and the output is the search results displayed in the user's browser.

[0544] Step 9:

[0545] The user selects the information they need from the displayed list of search results and checks the details. For example, they click on the subject of a specific email to read the full text and extract important points. The input data is the displayed search results, and the output data is the email content and chat history displayed in the detailed view.

[0546] (Application example 1)

[0547] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0548] Currently, security service inquiries and trouble report history management is distributed across multiple communication tools (email, chat, etc.), and there is an inadequate system for integrated management and search. As a result, information sharing between end users and support teams is delayed, making it difficult to respond quickly. Furthermore, when searching for related data, users often cannot specify appropriate search criteria, which often hinders efficient information retrieval. These issues need to be resolved.

[0549] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0550] In this invention, the server includes means for acquiring history data from various communication tools, means for converting the history data into a unified format, means for storing the data converted into the unified format in a database, means for searching the database for corresponding history data based on search criteria specified by a user, means for visualizing the searched history data to the user, means for acquiring history data using data collection software and interface software, converting the data into a unified format, and storing the data, means for performing search and visualization through an application compatible with the user's smart device, and means for recommending search criteria and automatically generating prompt sentences using a generative AI model. This allows for centralized management of distributed communication histories between end users and support teams, enabling rapid and appropriate information search and sharing.

[0551] "Various communication tools" is a general term for communication methods using different media, such as email, chat applications, and messaging platforms.

[0552] "Historical Data" means records of messages, emails, and other communications exchanged through various communication tools.

[0553] A "uniform format" is a standardized data format for converting different types of historical data into a consistent format.

[0554] "Database" means a centralized data management system for efficiently storing, searching, and managing historical data that has been converted into a unified format.

[0555] "Search criteria" are criteria such as keywords, date ranges, senders, and recipients that a user specifies to identify and search for specific historical data.

[0556] "Visualization" is the process of displaying retrieved historical data in a way that is easy for users to understand.

[0557] "Data collection software" is a program for automatically collecting historical data from various communication tools.

[0558] "Interface software" refers to the operating screens and applications that allow users to interact with the data collection software and database.

[0559] "Smart devices" are highly functional portable devices such as smartphones, tablets, smart glasses, and head-mounted displays.

[0560] A "generative AI model" is a computational model that uses artificial intelligence techniques to automatically perform specific tasks.

[0561] "Search criterion recommendation" is a function in which the generative AI model automatically suggests appropriate search criteria so that users can specify them.

[0562] "Automatic prompt generation" is the process of using a generative AI model to automatically generate commands that users use when searching or performing operations.

[0563] The present invention provides a system that uses a user's smart device to centrally manage, search, and visualize historical data of inquiries and trouble reports for security services. Specific embodiments of this system will be described below.

[0564] Specific methods for collecting data

[0565] server:

[0566] The server periodically retrieves historical data from various communication tools. For example, it retrieves emails from a mail server using the IMAP or POP3 protocol, and retrieves messages from chat tools using an API. A parser is used to convert the retrieved data into a unified format. In the case of email data, information such as the subject, body, sender / receiver, and date / time is retrieved, and in the case of chat data, information on the message content, sender, recipient, and timestamp is retrieved. The converted data is stored in a database. This database checks for duplicates and manages the data in a unified format.

[0567] Specific methods for data search

[0568] User:

[0569] Users access a dedicated search interface and enter search criteria such as search keywords, date ranges, and specific senders and recipients. The interface is provided through a smart device application. When the user presses the search button, the server receives the search request and generates an SQL query based on the criteria. The generated SQL query is used to search corresponding historical data from the database and retrieve search results. The search results are then sorted in chronological order or by relevance.

[0570] Specific display and operation methods

[0571] Device:

[0572] The application installed on the user's smart device displays the search results returned from the server on the user's screen. For emails, the search results display the subject, sender, recipient, and date and time, while for chats, a preview of the message is displayed. The user can click on individual items in the search results to view more information. Pressing the details button displays the full text of the relevant email or chat history.

[0573] Utilizing generative AI models and prompts

[0574] Generative AI models:

[0575] The server uses a generative AI model to provide a function that recommends search terms entered by the user. For example, if a user enters an ambiguous keyword, the server automatically suggests specific search terms related to that keyword. Furthermore, the server also automatically generates prompt sentences to help users perform searches efficiently.

[0576] Examples:

[0577] User A wants to investigate past correspondence history regarding "Project Y" and accesses the system's search interface. User A enters "Project Y" as a keyword and sets the date range to the past six months. When he presses the search button, the server searches the database for emails and chat history related to "Project Y," formats it, and returns it to User A's device. User A checks the list of search results displayed, clicks on the subject of a related email to read the full text, and extracts important points.

[0578] Example prompt sentence:

[0579] A user wants to search for past security-related incident history using specific keywords. When the user enters "Keyword: security incident" and "Date range: January 1, 2023 to December 31, 2023", generate Python code to retrieve the above historical data and display relevant results.

[0580] In this way, the proposed system quickly and efficiently integrates distributed communication histories, allowing users to easily obtain the information they need.

[0581] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0582] Step 1:

[0583] The server acquires historical data from various communication tools (e.g., mail servers, chat APIs). Specifically, it acquires email data using the IMAP or POP3 protocol, and message data using the chat tool's API. The input is raw data from each communication tool, and the output is the acquired raw data.

[0584] Step 2:

[0585] The server converts the acquired raw data into a unified format. For example, email data is broken down into items such as subject, body, sender, sender, and date and time, while chat data is broken down into message content, sender, receiver, and timestamp information. The input is raw data, and the output is data converted into a unified format.

[0586] Step 3:

[0587] The server saves the data converted into a unified format in the database. At the same time, it checks for duplicates and removes items that overlap with existing data. The input is unified format data, and the output is the data saved in the database.

[0588] Step 4:

[0589] A user accesses a dedicated search interface installed on a smart device and inputs search criteria such as search keywords, date range, sender / receiver, etc. The input is the search criteria specified by the user, and the output is a search request sent to the search interface.

[0590] Step 5:

[0591] The server receives a search request from a user, generates an SQL query based on the conditions, and searches the corresponding historical data from the database. The input is the search request from the user, and the output is the historical data in the database that matches the search conditions.

[0592] Step 6:

[0593] The server sorts the search results in chronological order or by relevance. The input is the searched history data, and the output is the sorted search results.

[0594] Step 7:

[0595] The server uses a generative AI model to automatically generate search query recommendations and prompts for users. The input is the initial search query and data in the database, and the output is the recommended search query and prompt.

[0596] Step 8:

[0597] The terminal displays the search results returned from the server. For emails, the search results show the subject, sender, recipient, and date and time, and for chats, a preview of the message is displayed. The input is the search results from the server, and the output is the search results in list format that is displayed on the user's screen.

[0598] Step 9:

[0599] Users can click on individual items from the displayed search results to check detailed information. Pressing the details button displays the full text of the relevant email or chat history. The input is the user's click, and the output is a popup or separate screen displaying detailed information.

[0600] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0601] This invention combines a system that acquires history data from various communication tools, converts it into a unified format, stores it in a database, searches for corresponding history data based on search conditions specified by the user, and visualizes it for the user with an emotion engine that recognizes the user's emotions. This system can adjust search results and change presentation methods based on the user's emotions.

[0602] Data collection

[0603] server:

[0604] 1. The server obtains historical data from various communication tools (e.g., mail servers, chat servers). Examples include obtaining new emails from a mail server using the IMAP or POP3 protocol, or obtaining messages using a chat tool's API.

[0605] 2. To convert the acquired data into a unified format, the subject, body, sender, sender, and date / time are extracted from email data, and the message content, sender, receiver, and timestamp information are extracted from chat data.

[0606] 3. Save the converted data in the database and check for duplicates.

[0607] Data Search

[0608] User:

[0609] 1. The user accesses a dedicated search interface and enters search criteria such as search keywords, a date range, or specific senders / receivers. The input interface uses text boxes, drop-down menus, and calendar widgets.

[0610] 2. Set the search conditions and press the search button.

[0611] server:

[0612] 1. The server receives a search request from a user and generates a SQL query, for example, specifying the keyword "Project X" and a specific date range.

[0613] 2. Run the generated SQL query against the database to retrieve search results, sorted chronologically and / or by relevance.

[0614] Emotion engine processing

[0615] server:

[0616] 1. Before formatting the search results, we use an emotion engine to recognize the user's emotions by analyzing the user's speech and input text when entering search criteria.

[0617] 2. Tailor search results based on perceived emotions, for example, prioritizing more important information if a user is feeling stressed.

[0618] 3. The presentation of search results will also change based on emotion, for example, results will be displayed in vibrant colors for positive emotions and muted colors for negative emotions.

[0619] Display and Operation

[0620] Device:

[0621] 1. The search results returned from the server are displayed on the user's screen. For emails, the results show the subject, sender, recipient, and date and time, and for chats, a preview of the message is displayed.

[0622] 2. Users can click on search results to learn more, displaying a specific email or chat history in a modal window or new page, with other related messages available as links.

[0623] Specific examples

[0624] scenario:

[0625] 1. User A accesses the system's search interface to investigate past response history regarding "Project Y."

[0626] 2. User A enters "Project Y" as a keyword and sets the date range to the past six months. The emotion engine recognizes from User A's voice and text that he or she is feeling stressed.

[0627] 3. When the search button is pressed, the server searches the database for emails and chat history related to "Project Y," prioritizes the most important information based on the stress level recognized by the emotion engine, and returns it to User A's device.

[0628] 4. User A reviews the list of search results, clicks on the subject of the relevant email to read the full text and extract the key points.

[0629] In this way, the proposed system takes into account the user's emotions, quickly and efficiently integrates distributed communication histories, and allows users to easily obtain the information they need.

[0630] The processing flow will be explained below.

[0631] Step 1: Get the data

[0632] server:

[0633] The server is configured to periodically retrieve new messages and historical data using the APIs or protocols of various communication tools (e.g., IMAP, POP3).

[0634] To retrieve new mail, the IMAP protocol is used to retrieve new mail from each user's inbox.

[0635] Call the chat tool's public API to get new messages from that tool.

[0636] Step 2: Transform the data

[0637] server:

[0638] The acquired data is temporarily saved and converted into a unified format (e.g., JSON format).

[0639] In the case of email data, information such as the subject, body, sender / receiver, date and time is extracted.

[0640] For chat data, message content, sender, recipient, and timestamp are extracted.

[0641] Step 3: Save your data

[0642] server:

[0643] The converted data is checked for duplicates and saved in the database.

[0644] Save new data to the database using INSERT operations and, if necessary, update operations.

[0645] Index the database to speed up later searches.

[0646] Step 4: Enter search criteria

[0647] User:

[0648] Users access a search interface and enter criteria such as search keywords, date ranges, and specific senders and recipients.

[0649] The interface allows you to enter search keywords into the text box and set a date range using the calendar widget.

[0650] Click the search button.

[0651] Step 5: Recognize emotions

[0652] server:

[0653] The emotion engine recognizes the user's emotions through text analysis of the user's input (keywords and comments) or, in the case of voice input, through voice analysis.

[0654] Data about the recognized emotions is temporarily stored for use in adjusting search queries and result display.

[0655] Step 6: Generating a search query

[0656] server:

[0657] It receives a search request from the user and generates an SQL query taking into account the results of the emotion engine.

[0658] For example, dynamically generate SQL statements specifying keywords related to "Project X" and a specific date range, and prioritize the importance of the results if the user's sentiment is negative.

[0659] Step 7: Database Search

[0660] server:

[0661] Run the generated SQL queries against the database to find relevant historical data.

[0662] Search results are sorted chronologically and by relevance based on user sentiment.

[0663] Step 8: Formatting the search results

[0664] server:

[0665] Format the search results and create a response to return to the user.

[0666] Generates JSON and HTML responses and includes additional information for each search result (e.g., email previews, chat message excerpts).

[0667] Based on the results of the emotion engine, the way search results are displayed (color tone, layout, etc.) is adjusted.

[0668] Step 9: Viewing search results

[0669] Device:

[0670] The search results returned from the server are displayed on the user's screen.

[0671] Results can be sorted chronologically or by relevance, and for emails, the subject, sender, sender, and date and time are displayed.

[0672] Step 10: View details and operate

[0673] User:

[0674] Users click on search result items to see more details.

[0675] View a specific email or chat history in a modal window or new page, with other related messages displayed as links.

[0676] Example 2

[0677] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0678] Conventional communication history search systems make it difficult for users to efficiently retrieve the information they need from the vast number of emails and chat histories. Furthermore, they are unable to prioritize information display based on the user's emotions or change the display format, so there is a need for improved user experience. Furthermore, processing search requests from multiple users simultaneously places a heavy load on the system, resulting in a decrease in system performance.

[0679] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0680] In this invention, the server includes means for acquiring communication history data from various information communication devices, means for converting the communication history data into a standard format, means for storing the data converted into the standard format in a storage device, means for searching the storage device for corresponding communication history data based on search criteria specified by a user, means for displaying the searched communication history data to the user, means for recognizing the user's emotion, means for adjusting the search results based on the recognized emotion, and means for changing the display format of the search results based on the recognized emotion. This enables prioritized display of information and change of the display format according to the user's emotion, allowing necessary information to be obtained quickly and efficiently. System performance is also improved when simultaneously processing search requests from multiple users.

[0681] An "information communication device" is a device that provides communication history data, such as an email server or chat server.

[0682] "Communication history data" refers to data that includes information such as the content of emails and chat messages, senders and receivers, and dates and times.

[0683] The "standard format" is a data format for converting communication history data in different formats into a unified format.

[0684] The "storage device" is a storage medium such as a database for storing communication history data.

[0685] "Search criteria" are search criteria such as keywords, date ranges, senders and recipients specified by the user.

[0686] The "display means" is an interface for visually presenting search results to the user.

[0687] "Emotion recognition" is a technology that analyzes and recognizes the emotions expressed by users when they enter search criteria.

[0688] "Search result tailoring" refers to changing the priorities of search results based on perceived user sentiment.

[0689] "Changing the display format" means changing the presentation method of search results according to the user's emotions.

[0690] This invention combines a system that acquires history data from various information and communication devices, converts it into a standard format, stores it in a storage device, searches for corresponding history data based on search conditions specified by the user, and visualizes it for the user with an emotion engine that recognizes the user's emotions. This system can adjust search results and change the presentation method based on the user's emotions.

[0691] The server first obtains communication history data from various information and communication devices (e.g., mail servers, chat servers). Specifically, it obtains new emails from the mail server using the IMAP or POP3 protocol, and then obtains messages using the chat tool's API. Specific software libraries called IMAPClient and Requests are used.

[0692] To convert the retrieved data into a standard format, extract the subject, body, sender, recipient, and date and time from email data, and extract the message content, sender, recipient, and timestamp information from chat data. To extract the subject from email data, use email.message_from_string(raw_email)['Subject'], and to extract the message content from chat data, use json.loads(response.text)['messages'].

[0693] Next, the converted data is saved to a storage device (database) and checked for duplicates. To save to the database, an SQL query (e.g., INSERT INTO communications (subject, body, sender, recipient, date) VALUES ...) is executed. A duplicate check is performed using an SQL query (e.g., SELECT COUNT() FROM communications WHERE ...).

[0694] Users access a specialized search interface and enter search criteria, such as search keywords, date ranges, specific senders and recipients, using text boxes, drop-down menus, and calendar widgets. When the user presses a search button, the search criteria are sent to the server.

[0695] Next, the server receives a search request from the user and generates an SQL query based on the search criteria, for example, sql_query = f"SELECT FROM communications WHERE subject LIKE '%{keyword}%' AND date BETWEEN '{start_date}' AND '{end_date}'". It executes this SQL query, retrieves the results, and sorts them by chronological order or relevance.

[0696] Furthermore, the server uses an emotion engine to recognize the user's emotions before formatting the search results. To recognize the user's emotions, it uses an emotion recognition API such as emotion = analyze_emotion(user_input). Based on the recognized emotion, it adjusts the search results. For example, if stress is recognized, it reorders the search results using the method results = prioritize_important_information(results). The presentation method of the search results is also changed accordingly.

[0697] Search results are displayed on the user's device. For emails, the results show the subject, sender, recipient, and date and time, and for chats, a preview of the message is displayed. Users can click on a result to view more information. The history of a specific email or chat can be displayed in a modal window or on a new page, with other related messages displayed as links.

[0698] Specific examples

[0699] 1. User A accesses the system's search interface to investigate past response history regarding "Project Y."

[0700] 2. User A enters "Project Y" as a keyword and sets the date range to the past six months. The emotion engine recognizes from User A's voice and text that he or she is feeling stressed.

[0701] 3. When the search button is pressed, the server searches the database for emails and chat history related to "Project Y," prioritizes the most important information based on the stress level recognized by the emotion engine, and returns it to User A's device.

[0702] 4. User A reviews the list of search results, clicks on the subject of the relevant email to read the full text and extract the key points.

[0703] Prompt Sentence Examples

[0704] I want to investigate the past history of projects related to Project Y. The emotion engine should recognize the user's emotions and prioritize the most important information. Then, display the search results based on color and presentation style.

[0705] In this way, the proposed system takes into account the user's emotions, quickly and efficiently integrates distributed communication histories, and allows users to easily obtain the information they need.

[0706] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0707] Step 1:

[0708] server:

[0709] Input: Connection information and authentication information obtained from various information and communication devices.

[0710] Operation: The server connects to various information and communication devices (e.g., mail servers, chat servers) and acquires communication history data. Specifically, it acquires new emails from the mail server using the IMAP or POP3 protocol, and acquires messages using the chat tool's API.

[0711] Output: Acquired communication history data (e.g., content of emails and chat messages, sender / receiver, date and time, etc.).

[0712] Step 2:

[0713] server:

[0714] Input: Acquired communication history data (e.g., raw email data, chat message data).

[0715] Operation: The server converts the acquired communication history data into a standard format. From email data, it extracts the subject, body, sender, recipient, and date and time, and from chat data, it extracts the message content, sender, recipient, and timestamp information. Specifically, to extract the subject from email data, it uses email.message_from_string(raw_email)['Subject'], and to extract the message content from chat data, it uses json.loads(response.text)['messages'].

[0716] Output: Communication history data converted into a standard format.

[0717] Step 3:

[0718] server:

[0719] Input: Communication history data converted into a standard format.

[0720] Operation: The server converts the communication history data into a standard format, stores it in a storage device (database), and checks for duplicates. Specifically, to store it in the database, it executes the SQL query INSERT INTO communications (subject, body, sender, recipient, date) VALUES ..., and to check for duplicates, it uses the SQL query SELECT COUNT() FROM communications WHERE ....

[0721] Output: Saved communication history data and whether there are any duplicates.

[0722] Step 4:

[0723] User:

[0724] Input: Access the search interface, search criteria such as search keywords, date range, or specific senders / receivers.

[0725] How it works: Users access a dedicated search interface and enter search criteria using text boxes, drop-down menus, and calendar widgets.

[0726] Output: The search criteria that was set.

[0727] Step 5:

[0728] server:

[0729] Input: The search criteria submitted by the user.

[0730] How it works: The server receives a search request from a user and generates an SQL query based on the search criteria. Specifically, it generates an SQL query in the format sql_query = f"SELECT FROM communications WHERE subject LIKE '%{keyword}%' AND date BETWEEN '{start_date}' AND '{end_date}'".

[0731] Output: The generated SQL query.

[0732] Step 6:

[0733] server:

[0734] Input: The generated SQL query.

[0735] How it works: The server executes an SQL query and retrieves search results. The server then executes an SQL query to sort the results in chronological order or relevance order, such as ORDER BY date DESC.

[0736] Output: Sorted search results.

[0737] Step 7:

[0738] server:

[0739] Input: Sorted search results.

[0740] Operation: The server uses the emotion engine to recognize the user's emotions. Specifically, it analyzes the text and voice input of the user's search criteria and calls the emotion recognition API, emotion = analyze_emotion(user_input).

[0741] Output: Perceived user sentiment.

[0742] Step 8:

[0743] server:

[0744] Input: Perceived user sentiment, sorted search results.

[0745] How it works: The server tailors search results based on the perceived emotion. For example, if stress is perceived, it reorders the search results to prioritize more important information using the method results = prioritize_important_information(results).

[0746] Output: Refined search results.

[0747] Step 9:

[0748] server:

[0749] Input: Perceived user sentiment, adjusted search results.

[0750] How it works: The server changes the presentation of search results based on the recognized emotion, for example, presenting vibrant colors for positive emotions and calm colors for negative emotions, using theme-changing methods like if emotion == 'positive': set_vibrant_theme() else: set_calm_theme().

[0751] Output: The format in which the final search results are displayed.

[0752] Step 10:

[0753] Device:

[0754] Input: The format in which the final search results will be displayed.

[0755] What it does: Search results returned by the server are displayed on the user's screen. For emails, the search results show the subject, sender / receiver, and date and time, and for chats, a preview of the message is displayed.

[0756] Output: Search results displayed on the user's screen.

[0757] Step 11:

[0758] User:

[0759] Input: The search result item the user selects on the screen.

[0760] What it does: Users click on a search result item to learn more, or view a specific email or chat history in a modal window or new page, with other related messages available as links.

[0761] Output: A modal window or new page with more information.

[0762] (Application example 2)

[0763] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0764] Conventional food delivery systems only provide information based on order history, support interaction history, and review comments without considering the user's emotions. As a result, when a user is stressed or in a particular emotional state, appropriate information is not provided, resulting in a decrease in user satisfaction.

[0765] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0766] In this invention, the server includes means for acquiring history data from various communication means, means for converting the history data into a unified format, means for storing the data converted into the unified format in a data storage means, means for searching for corresponding history data from the data storage means based on search conditions specified by a user, means for visualizing the searched history data to the user, an emotion recognition engine for analyzing the user's emotions, means for adjusting search results based on the emotions, and means for changing the presentation format based on the emotions, thereby making it possible to provide appropriate information according to the user's emotions.

[0767] "Various communication means" refers to means for exchanging information via multiple different communication protocols and devices.

[0768] "History data" refers to information about past interactions with a user, and primarily includes order history, support interaction history, review comments, and the like.

[0769] A "uniform format" is a standardized data format for consistently handling data acquired in different formats.

[0770] "Data storage means" refers to a means for saving acquired historical data, and typically includes databases and cloud storage.

[0771] "Search conditions" are key information that a user specifies when retrieving specific data, and include keywords, date ranges, specific senders and receivers, and the like.

[0772] The "visualization means" is a means for visually displaying search results to the user, such as a graphical user interface.

[0773] An "emotion recognition engine" is an engine that has the function of analyzing a user's emotions, and is a technology that analyzes emotions from a user's speech or text input.

[0774] The "means for adjusting search results based on emotions" refers to a means for changing the content and display order of search results according to the user's emotions analyzed by the emotion recognition engine.

[0775] The "means for changing the presentation method based on emotions" is a means for adjusting the display method of search results and the UI theme based on the user's emotions.

[0776] The embodiments for carrying out the present invention are as follows.

[0777] Data collection and storage

[0778] The server has a means of acquiring historical data from various communication methods. Specifically, data such as email, chat messages, and order history is acquired using multiple APIs and communication protocols (e.g., IMAP, POP3, HTTP). The acquired historical data is converted into a unified format. For example, the subject, body, sender, sender, and date / time are extracted from email, and the message content, sender, receiver, and timestamp information are extracted from chat messages. The converted data is then saved in an SQLite database, which serves as the data storage method. A check for duplicate data is also performed when saving the data.

[0779] Data Search

[0780] A user accesses a dedicated search interface and enters search criteria. For example, in a food delivery service application, a user can specify keywords and a date range for order history. The server receives the user's search request and generates an SQL query. For example, an SQL statement specifying the keyword "order" and a specific date range is generated. The generated SQL query is executed against the database to retrieve the corresponding history data.

[0781] Sentiment Analysis Engine

[0782] When a user enters search criteria, the emotion recognition engine analyzes the user's emotions. This includes the text the user enters and speech using speech recognition technology. This determines the emotional state as positive, negative, or neutral. The emotion recognition engine uses TextBlob and other natural language processing libraries.

[0783] Tailoring and displaying search results

[0784] The server adjusts search results based on the emotions analyzed by the emotion recognition engine. For example, if the user is feeling stressed, it prioritizes displaying information that is of high importance. It also changes the presentation style based on the emotion, applying a brightly colored design for positive emotions and a more subdued design for negative emotions. The adjusted search results are visualized on the user's device. The results may include, for example, order history details or support interaction history, displayed in chronological order or by relevance.

[0785] Specific examples

[0786] If a user types "my order is late," the server uses an emotion recognition engine to analyze negative emotions. It then adjusts search results, prioritizing important information and visualizing the results in calming colors. Users can then respond quickly based on the displayed results, reducing stress.

[0787] Prompt Sentence Examples

[0788] Use the Food Delivery Emotional Assistant to investigate my pizza order history. I'm stressed because my order is late.

[0789] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0790] Step 1:

[0791] The server acquires historical data from various communication methods. Specifically, it collects new orders, chat messages, and review comments via APIs such as mail servers and chat servers. It receives response data from the API as input and generates a list of the acquired raw data as output.

[0792] Step 2:

[0793] The server converts the acquired historical data into a unified format. Specifically, it extracts the subject, body, sender, recipient, and date and time from email data, and the message content, sender, recipient, and timestamp information from chat messages. It receives a list of raw data as input and generates a list of standardized data as output.

[0794] Step 3:

[0795] The server saves the converted data in a database, which is a data storage means. Specifically, it inserts the data into an SQLite database and also checks for duplicates. It receives a list of standardized data as input and generates the number of records of the data stored in the database as output.

[0796] Step 4:

[0797] A user accesses a dedicated search interface and enters search criteria, such as search keywords and date ranges. The system takes the search criteria specified by the user as input and generates a query based on the search criteria as output.

[0798] Step 5:

[0799] The server generates an SQL query based on the search criteria it receives. Specifically, it creates an SQL statement that includes search keywords and date ranges. It takes the search criteria specified by the user as input and generates an executable SQL statement as output.

[0800] Step 6:

[0801] The server executes the generated SQL query against the database to search for the corresponding historical data. Specifically, it retrieves the relevant records from the database and sorts them in chronological order or by relevance. It receives the SQL statement as input and generates a list of search results as output.

[0802] Step 7:

[0803] The server uses an emotion recognition engine to analyze the user's emotions. Specifically, it analyzes the text and voice input by the user and determines whether the emotion is positive, negative, or neutral. It receives the user's text and voice data as input and generates the type of emotion as output.

[0804] Step 8:

[0805] The server adjusts search results based on the user's emotions. Specifically, it sorts the search results to prioritize the most important information. As input, it receives the emotion type and a list of search results, and as output, it generates a list of adjusted search results.

[0806] Step 9:

[0807] The server displays the adjusted search results in an appropriate presentation style, specifically, applying a vibrant design for positive emotions and a subdued design for negative emotions. The server receives the adjusted search results list as input and generates a search result screen to be displayed on the user's device as output.

[0808] Step 10:

[0809] The user can interact with the displayed search results and view more information. For example, they can click on a specific order history or support interaction history to view its contents. The system takes the displayed search results as input and displays a modal window or a new page with more information as output.

[0810] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0811] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0812] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0813] [Third embodiment]

[0814] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0815] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0816] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0817] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0818] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0819] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0820] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0821] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0822] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0823] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0824] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0825] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0826] This invention provides a specific method for implementing a system that acquires historical data from various communication tools, converts it into a unified format, stores it in a database, searches for corresponding historical data based on search conditions specified by the user, and visualizes it for the user.

[0827] Data collection

[0828] server:

[0829] 1. The server periodically retrieves historical data from each communication tool (e.g., mail server, chat server). This includes retrieving new emails from the mail server using the IMAP or POP3 protocol, and retrieving messages using the chat tool's API.

[0830] 2. A parser is used to convert the retrieved data into a unified format. For example, for email data, information such as the subject, body, sender, recipient, and date and time is retrieved, and for chat data, message content, sender, recipient, and timestamp information is retrieved.

[0831] 3. The converted data is saved in the database. This data is checked for duplicates and saved in the database in a unified format.

[0832] Data Search

[0833] User:

[0834] 1. The user accesses a dedicated search interface and enters search criteria such as search keywords, a date range, or specific senders / receivers. The interface is presented using text boxes, drop-down menus, and calendar widgets.

[0835] 2. Set the search conditions and press the search button.

[0836] server:

[0837] 1. The server receives a search request from the user and generates a SQL query based on the criteria. For example, if you specify the keyword "Project X" and a specific date range, it generates a SQL statement to search for historical data that matches those criteria.

[0838] 2. The generated SQL query is executed against the database to retrieve search results, which are then sorted by chronological order and relevance.

[0839] Display and Operation

[0840] Device:

[0841] 1. The search results returned from the server are displayed on the user's screen. For emails, the search results include the subject, sender, recipient, and date and time, and for chats, a preview of the message is displayed.

[0842] 2. Users can click on individual search results to view more information. For example, clicking the Details button will open a modal window or new page with the full text of the email or chat history.

[0843] Specific examples

[0844] scenario:

[0845] 1. User A wants to investigate past response history regarding "Project Y" and accesses the system's search interface.

[0846] 2. User A enters "Project Y" as a keyword and sets the date range to the past 6 months.

[0847] 3. When the search button is pressed, the server searches the database for emails and chat history related to "Project Y," formats it, and returns it to User A's device.

[0848] 4. User A reviews the list of search results, clicks on the subject of the relevant email to read the full text and extract the key points.

[0849] In this way, the proposed system quickly and efficiently integrates distributed communication histories, allowing users to easily obtain the information they need.

[0850] The processing flow will be explained below.

[0851] Step 1: Get the data

[0852] server:

[0853] The server is configured to periodically retrieve new messages and historical data using the APIs or protocols of various communication tools (e.g., IMAP, POP3).

[0854] To retrieve new mail, the IMAP protocol is used to retrieve new mail from each user's inbox.

[0855] Call the chat tool's public API to get new messages from that tool.

[0856] Step 2: Transform the data

[0857] server:

[0858] The acquired data is temporarily saved and converted into a unified format (e.g., JSON format).

[0859] In the case of email data, information such as the subject, body, sender / receiver, date and time is extracted.

[0860] For chat data, message content, sender, recipient, and timestamp are extracted.

[0861] Step 3: Save your data

[0862] server:

[0863] The converted data is checked for duplicates and saved in the database.

[0864] Save new data to the database using INSERT operations and, if necessary, update operations.

[0865] Index the database to speed up later searches.

[0866] Step 4: Enter search criteria

[0867] User:

[0868] Users access a search interface and enter criteria such as search keywords, date ranges, and specific senders and recipients.

[0869] The interface allows you to enter search keywords into the text box and set a date range using the calendar widget.

[0870] Click the search button.

[0871] Step 5: Generating a search query

[0872] server:

[0873] It receives search requests from users and generates SQL queries based on the conditions.

[0874] For example, dynamically generate a SQL statement specifying keywords and a date range related to "Project X."

[0875] The generated query is sent to the database.

[0876] Step 6: Database Search

[0877] server:

[0878] Run the generated SQL queries against the database to find relevant historical data.

[0879] Search results are sorted chronologically and by relevance.

[0880] Step 7: Formatting the search results

[0881] server:

[0882] Format the search results and create a response to return to the user.

[0883] Generates JSON and HTML responses and includes additional information for each search result (e.g., email previews, chat message excerpts).

[0884] Step 8: Viewing search results

[0885] Device:

[0886] The search results returned from the server are displayed on the user's screen.

[0887] Results can be sorted chronologically or by relevance, and for emails, the subject, sender, sender, and date and time are displayed.

[0888] Step 9: View details and take action

[0889] User:

[0890] Users click on search result items to see more details.

[0891] View a specific email or chat history in a modal window or new page, with other related messages displayed as links.

[0892] Example 1

[0893] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0894] Conventional information retrieval systems face the problem of difficulty in centrally managing historical information from various information transmission tools, making it difficult to quickly search for and display specific information. Furthermore, performance often declines when processing search requests from multiple users in parallel. Another issue is the lack of a means to effectively sort and display search results.

[0895] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0896] In this invention, the server includes means for acquiring history information from various information transmission tools, means for converting the history information into a unified format, means for storing the information converted into the unified format in a data warehouse, means for searching the data warehouse for corresponding history information based on search conditions specified by a user, means for visualizing the searched history information to the user, means for periodically acquiring history information, means for checking for duplication and storing the information, means for accessing a dedicated search interface, means for inputting search keywords, date ranges, and specific senders and receivers, means for generating a query language based on the conditions, and means for sorting the acquired data in chronological order or by relevance. This makes it possible to efficiently collect, manage, and search history information from distributed information transmission tools and quickly respond to search requests from multiple users.

[0897] "Information communication tools" refers to communication methods such as email, chat, and messaging applications.

[0898] "History information" refers to messages and data sent and received through various information transmission tools, as well as related metadata.

[0899] "Unified format" refers to a format that converts data obtained from different information delivery tools into a consistent format.

[0900] "Data warehouse" refers to a database system designed to store and manage acquired historical information and enable efficient search and extraction.

[0901] "Search conditions" refer to conditions such as keywords, date ranges, senders, and recipients that a user specifies when searching for specific information.

[0902] "Duplicate check" refers to a confirmation process to prevent the same history information from being saved twice.

[0903] "Search interface" refers to a user interface that allows a user to input search criteria and view search results.

[0904] "Query language" refers to a query language (e.g., SQL) used to retrieve data from a database under specific conditions.

[0905] "Chronological order" refers to the order in which data is arranged in the order in which it occurred.

[0906] "Sort by relevance" refers to the order in which data is arranged in descending order of relevance to the search criteria.

[0907] This invention provides a specific method for implementing a system that acquires historical information from various information transmission tools, converts it into a unified format, stores it in a data warehouse, searches for corresponding historical information based on search conditions specified by the user, and visualizes it for the user.

[0908] First, the server periodically collects historical information from multiple information transmission tools. It retrieves emails from the mail server using the IMAP or POP3 protocol, and retrieves messages from the chat tool via API. It uses a cron job to check for new data every hour. The collected data is temporarily stored in local storage.

[0909] The server then runs a parser to convert the collected history information into a unified format. For example, a Python script is used to extract information such as the subject, body, sender, recipient, and date and time from emails and convert them into JSON objects. Similarly, chat messages are converted into a unified format, with information such as the message content, sender, recipient, and timestamp.

[0910] The server checks for duplicates of the converted data before storing it in a data warehouse (e.g., MySQL, PostgreSQL). Message IDs and timestamps are used to check for duplicates. Data stored in the data warehouse is designed to be efficiently searchable and retrievalable.

[0911] Users access a dedicated search interface and enter search criteria (e.g., search keywords, date ranges, specific senders / receivers). The search interface is provided using text boxes, drop-down menus, and calendar widgets. When a search button is pressed, the server receives the search request and generates an SQL query based on the received criteria.

[0912] For example, if the conditions "Project Y" and "Last 6 months" are specified, the server generates an SQL query such as "SELECT FROM communication_logs WHERE body LIKE '%ProjectY%' AND timestamp BETWEEN 'Start Date' AND 'End Date'". The server then queries the database and retrieves the search results, which are sorted chronologically and by relevance.

[0913] The device displays the search results returned by the server on the user's screen. For emails, the subject, sender, recipient, and date and time are displayed, and for chats, a preview of the message is displayed. The user can click on individual search results to view more information.

[0914] Specific examples

[0915] scenario:

[0916] 1. User A wants to investigate past response history regarding "Project Y" and accesses the system's search interface.

[0917] 2. User A enters "Project Y" as a keyword and sets the date range to the past 6 months.

[0918] 3. When the search button is pressed, the server searches the data warehouse for emails and chat history related to "Project Y," formats the data, and returns it to User A's device.

[0919] 4. User A reviews the list of search results, clicks on the subject of the relevant email to read the full text and extract the key points.

[0920] Example prompts for generative AI models

[0921] "Please explain the algorithm for a system that converts historical information obtained from various information transmission tools into a unified format and stores it in a data warehouse. This includes the ability for users to search for specific keywords or date ranges."

[0922] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0923] Step 1:

[0924] The server obtains history information from the information transmission tool. For example, it obtains messages from the mail server using the IMAP or POP3 protocol, and from the chat tool via an API. Input data includes authentication information for the mail server and the API key for the chat tool. Based on this, new emails and messages are temporarily saved in local storage.

[0925] Step 2:

[0926] The server reads the history information from the temporary storage folder and converts it into a unified format. The input data is history information files in JSON and CSV format. Specifically, a Python script is executed to extract the email subject, body, sender / receiver, date and time, etc., and convert them into a unified JSON object. The output data is a unified JSON object containing the subject, body, sender, recipient, and timestamp information.

[0927] Step 3:

[0928] The server checks for duplicates in the converted data before storing it in the data warehouse. The input data is a unified JSON object. An SQL statement such as "INSERT INTO communication_logs (subject, body, sender, receiver, timestamp) VALUES (...) ON DUPLICATE KEY UPDATE ..." is executed against the database. The output is accurate historical information stored in the data warehouse.

[0929] Step 4:

[0930] Users access a dedicated search interface and enter search criteria. The input data can include search keywords, a date range, and specific senders and recipients. Based on this, the search interface receives the entered criteria and displays them on the screen. The output is the received search criteria.

[0931] Step 5:

[0932] The server receives a search request from the user and generates an SQL query based on the conditions. For example, if "Project Y" and "Last 6 months" are specified, it generates an SQL query such as "SELECT FROM communication_logs WHERE body LIKE '%ProjectY%' AND timestamp BETWEEN 'Start date' AND 'End date'". The input data is the search conditions, and the output data is the generated SQL query.

[0933] Step 6:

[0934] The server executes the generated SQL query and retrieves the search results from the data warehouse. This involves executing the SQL query against the database and retrieving the relevant data. The input data is the SQL query, and the output data is the dataset of the search results.

[0935] Step 7:

[0936] The server sorts and formats the search results in chronological order or relevance order. For example, it sorts the list of search results by timestamp. The input data is the dataset of search results, and the output data is the sorted dataset.

[0937] Step 8:

[0938] The device displays the search results returned by the server on the user's screen: for emails, the subject, sender, recipient, and date and time are displayed, and for chats, a preview of the message is displayed. The input data is a sorted dataset, and the output is the search results displayed in the user's browser.

[0939] Step 9:

[0940] The user selects the information they need from the displayed list of search results and checks the details. For example, they click on the subject of a specific email to read the full text and extract important points. The input data is the displayed search results, and the output data is the email content and chat history displayed in the detailed view.

[0941] (Application example 1)

[0942] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0943] Currently, security service inquiries and trouble report history management is distributed across multiple communication tools (email, chat, etc.), and there is an inadequate system for integrated management and search. As a result, information sharing between end users and support teams is delayed, making it difficult to respond quickly. Furthermore, when searching for related data, users often cannot specify appropriate search criteria, which often hinders efficient information retrieval. These issues need to be resolved.

[0944] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0945] In this invention, the server includes means for acquiring history data from various communication tools, means for converting the history data into a unified format, means for storing the data converted into the unified format in a database, means for searching the database for corresponding history data based on search criteria specified by a user, means for visualizing the searched history data to the user, means for acquiring history data using data collection software and interface software, converting the data into a unified format, and storing the data, means for performing search and visualization through an application compatible with the user's smart device, and means for recommending search criteria and automatically generating prompt sentences using a generative AI model. This allows for centralized management of distributed communication histories between end users and support teams, enabling rapid and appropriate information search and sharing.

[0946] "Various communication tools" is a general term for communication methods using different media, such as email, chat applications, and messaging platforms.

[0947] "Historical Data" means records of messages, emails, and other communications exchanged through various communication tools.

[0948] A "uniform format" is a standardized data format for converting different types of historical data into a consistent format.

[0949] "Database" means a centralized data management system for efficiently storing, searching, and managing historical data that has been converted into a unified format.

[0950] "Search criteria" are criteria such as keywords, date ranges, senders, and recipients that a user specifies to identify and search for specific historical data.

[0951] "Visualization" is the process of displaying retrieved historical data in a way that is easy for users to understand.

[0952] "Data collection software" is a program for automatically collecting historical data from various communication tools.

[0953] "Interface software" refers to the operating screens and applications that allow users to interact with the data collection software and database.

[0954] "Smart devices" are highly functional portable devices such as smartphones, tablets, smart glasses, and head-mounted displays.

[0955] A "generative AI model" is a computational model that uses artificial intelligence techniques to automatically perform specific tasks.

[0956] "Search criterion recommendation" is a function in which the generative AI model automatically suggests appropriate search criteria so that users can specify them.

[0957] "Automatic prompt generation" is the process of using a generative AI model to automatically generate commands that users use when searching or performing operations.

[0958] The present invention provides a system that uses a user's smart device to centrally manage, search, and visualize historical data of inquiries and trouble reports for security services. Specific embodiments of this system will be described below.

[0959] Specific methods for collecting data

[0960] server:

[0961] The server periodically retrieves historical data from various communication tools. For example, it retrieves emails from a mail server using the IMAP or POP3 protocol, and retrieves messages from chat tools using an API. A parser is used to convert the retrieved data into a unified format. In the case of email data, information such as the subject, body, sender / receiver, and date / time is retrieved, and in the case of chat data, information on the message content, sender, recipient, and timestamp is retrieved. The converted data is stored in a database. This database checks for duplicates and manages the data in a unified format.

[0962] Specific methods for data search

[0963] User:

[0964] Users access a dedicated search interface and enter search criteria such as search keywords, date ranges, and specific senders and recipients. The interface is provided through a smart device application. When the user presses the search button, the server receives the search request and generates an SQL query based on the criteria. The generated SQL query is used to search corresponding historical data from the database and retrieve search results. The search results are then sorted in chronological order or by relevance.

[0965] Specific display and operation methods

[0966] Device:

[0967] The application installed on the user's smart device displays the search results returned from the server on the user's screen. For emails, the search results display the subject, sender, recipient, and date and time, while for chats, a preview of the message is displayed. The user can click on individual items in the search results to view more information. Pressing the details button displays the full text of the relevant email or chat history.

[0968] Utilizing generative AI models and prompts

[0969] Generative AI models:

[0970] The server uses a generative AI model to provide a function that recommends search terms entered by the user. For example, if a user enters an ambiguous keyword, the server automatically suggests specific search terms related to that keyword. Furthermore, the server also automatically generates prompt sentences to help users perform searches efficiently.

[0971] Examples:

[0972] User A wants to investigate past correspondence history regarding "Project Y" and accesses the system's search interface. User A enters "Project Y" as a keyword and sets the date range to the past six months. When he presses the search button, the server searches the database for emails and chat history related to "Project Y," formats it, and returns it to User A's device. User A checks the list of search results displayed, clicks on the subject of a related email to read the full text, and extracts important points.

[0973] Example prompt sentence:

[0974] A user wants to search for past security-related incident history using specific keywords. When the user enters "Keyword: security incident" and "Date range: January 1, 2023 to December 31, 2023", generate Python code to retrieve the above historical data and display relevant results.

[0975] In this way, the proposed system quickly and efficiently integrates distributed communication histories, allowing users to easily obtain the information they need.

[0976] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0977] Step 1:

[0978] The server acquires historical data from various communication tools (e.g., mail servers, chat APIs). Specifically, it acquires email data using the IMAP or POP3 protocol, and message data using the chat tool's API. The input is raw data from each communication tool, and the output is the acquired raw data.

[0979] Step 2:

[0980] The server converts the acquired raw data into a unified format. For example, email data is broken down into items such as subject, body, sender, sender, and date and time, while chat data is broken down into message content, sender, receiver, and timestamp information. The input is raw data, and the output is data converted into a unified format.

[0981] Step 3:

[0982] The server saves the data converted into a unified format in the database. At the same time, it checks for duplicates and removes items that overlap with existing data. The input is unified format data, and the output is the data saved in the database.

[0983] Step 4:

[0984] A user accesses a dedicated search interface installed on a smart device and inputs search criteria such as search keywords, date range, sender / receiver, etc. The input is the search criteria specified by the user, and the output is a search request sent to the search interface.

[0985] Step 5:

[0986] The server receives a search request from a user, generates an SQL query based on the conditions, and searches the corresponding historical data from the database. The input is the search request from the user, and the output is the historical data in the database that matches the search conditions.

[0987] Step 6:

[0988] The server sorts the search results in chronological order or by relevance. The input is the searched history data, and the output is the sorted search results.

[0989] Step 7:

[0990] The server uses a generative AI model to automatically generate search query recommendations and prompts for users. The input is the initial search query and data in the database, and the output is the recommended search query and prompt.

[0991] Step 8:

[0992] The terminal displays the search results returned from the server. For emails, the search results show the subject, sender, recipient, and date and time, and for chats, a preview of the message is displayed. The input is the search results from the server, and the output is the search results in list format that is displayed on the user's screen.

[0993] Step 9:

[0994] Users can click on individual items from the displayed search results to check detailed information. Pressing the details button displays the full text of the relevant email or chat history. The input is the user's click, and the output is a popup or separate screen displaying detailed information.

[0995] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0996] This invention combines a system that acquires history data from various communication tools, converts it into a unified format, stores it in a database, searches for corresponding history data based on search conditions specified by the user, and visualizes it for the user with an emotion engine that recognizes the user's emotions. This system can adjust search results and change presentation methods based on the user's emotions.

[0997] Data collection

[0998] server:

[0999] 1. The server obtains historical data from various communication tools (e.g., mail servers, chat servers). Examples include obtaining new emails from a mail server using the IMAP or POP3 protocol, or obtaining messages using a chat tool's API.

[1000] 2. To convert the acquired data into a unified format, the subject, body, sender, sender, and date / time are extracted from email data, and the message content, sender, receiver, and timestamp information are extracted from chat data.

[1001] 3. Save the converted data in the database and check for duplicates.

[1002] Data Search

[1003] User:

[1004] 1. The user accesses a dedicated search interface and enters search criteria such as search keywords, a date range, or specific senders / receivers. The input interface uses text boxes, drop-down menus, and calendar widgets.

[1005] 2. Set the search conditions and press the search button.

[1006] server:

[1007] 1. The server receives a search request from a user and generates a SQL query, for example, specifying the keyword "Project X" and a specific date range.

[1008] 2. Run the generated SQL query against the database to retrieve search results, sorted chronologically and / or by relevance.

[1009] Emotion engine processing

[1010] server:

[1011] 1. Before formatting the search results, we use an emotion engine to recognize the user's emotions by analyzing the user's speech and input text when entering search criteria.

[1012] 2. Tailor search results based on perceived emotions, for example, prioritizing more important information if a user is feeling stressed.

[1013] 3. The presentation of search results will also change based on emotion, for example, results will be displayed in vibrant colors for positive emotions and muted colors for negative emotions.

[1014] Display and Operation

[1015] Device:

[1016] 1. The search results returned from the server are displayed on the user's screen. For emails, the results show the subject, sender, recipient, and date and time, and for chats, a preview of the message is displayed.

[1017] 2. Users can click on search results to learn more, displaying a specific email or chat history in a modal window or new page, with other related messages available as links.

[1018] Specific examples

[1019] scenario:

[1020] 1. User A accesses the system's search interface to investigate past response history regarding "Project Y."

[1021] 2. User A enters "Project Y" as a keyword and sets the date range to the past six months. The emotion engine recognizes from User A's voice and text that he or she is feeling stressed.

[1022] 3. When the search button is pressed, the server searches the database for emails and chat history related to "Project Y," prioritizes the most important information based on the stress level recognized by the emotion engine, and returns it to User A's device.

[1023] 4. User A reviews the list of search results, clicks on the subject of the relevant email to read the full text and extract the key points.

[1024] In this way, the proposed system takes into account the user's emotions, quickly and efficiently integrates distributed communication histories, and allows users to easily obtain the information they need.

[1025] The processing flow will be explained below.

[1026] Step 1: Get the data

[1027] server:

[1028] The server is configured to periodically retrieve new messages and historical data using the APIs or protocols of various communication tools (e.g., IMAP, POP3).

[1029] To retrieve new mail, the IMAP protocol is used to retrieve new mail from each user's inbox.

[1030] Call the chat tool's public API to get new messages from that tool.

[1031] Step 2: Transform the data

[1032] server:

[1033] The acquired data is temporarily saved and converted into a unified format (e.g., JSON format).

[1034] In the case of email data, information such as the subject, body, sender / receiver, date and time is extracted.

[1035] For chat data, message content, sender, recipient, and timestamp are extracted.

[1036] Step 3: Save your data

[1037] server:

[1038] The converted data is checked for duplicates and saved in the database.

[1039] Save new data to the database using INSERT operations and, if necessary, update operations.

[1040] Index the database to speed up later searches.

[1041] Step 4: Enter search criteria

[1042] User:

[1043] Users access a search interface and enter criteria such as search keywords, date ranges, and specific senders and recipients.

[1044] The interface allows you to enter search keywords into the text box and set a date range using the calendar widget.

[1045] Click the search button.

[1046] Step 5: Recognize emotions

[1047] server:

[1048] The emotion engine recognizes the user's emotions through text analysis of the user's input (keywords and comments) or, in the case of voice input, through voice analysis.

[1049] Data about the recognized emotions is temporarily stored for use in adjusting search queries and result display.

[1050] Step 6: Generating a search query

[1051] server:

[1052] It receives a search request from the user and generates an SQL query taking into account the results of the emotion engine.

[1053] For example, dynamically generate SQL statements specifying keywords related to "Project X" and a specific date range, and prioritize the importance of the results if the user's sentiment is negative.

[1054] Step 7: Database Search

[1055] server:

[1056] Run the generated SQL queries against the database to find relevant historical data.

[1057] Search results are sorted chronologically and by relevance based on user sentiment.

[1058] Step 8: Formatting the search results

[1059] server:

[1060] Format the search results and create a response to return to the user.

[1061] Generates JSON and HTML responses and includes additional information for each search result (e.g., email previews, chat message excerpts).

[1062] Based on the results of the emotion engine, the way search results are displayed (color tone, layout, etc.) is adjusted.

[1063] Step 9: Viewing search results

[1064] Device:

[1065] The search results returned from the server are displayed on the user's screen.

[1066] Results can be sorted chronologically or by relevance, and for emails, the subject, sender, sender, and date and time are displayed.

[1067] Step 10: View details and operate

[1068] User:

[1069] Users click on search result items to see more details.

[1070] View a specific email or chat history in a modal window or new page, with other related messages displayed as links.

[1071] Example 2

[1072] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1073] Conventional communication history search systems make it difficult for users to efficiently retrieve the information they need from the vast number of emails and chat histories. Furthermore, they are unable to prioritize information display based on the user's emotions or change the display format, so there is a need for improved user experience. Furthermore, processing search requests from multiple users simultaneously places a heavy load on the system, resulting in a decrease in system performance.

[1074] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1075] In this invention, the server includes means for acquiring communication history data from various information communication devices, means for converting the communication history data into a standard format, means for storing the data converted into the standard format in a storage device, means for searching the storage device for corresponding communication history data based on search criteria specified by a user, means for displaying the searched communication history data to the user, means for recognizing the user's emotion, means for adjusting the search results based on the recognized emotion, and means for changing the display format of the search results based on the recognized emotion. This enables prioritized display of information and change of the display format according to the user's emotion, allowing necessary information to be obtained quickly and efficiently. System performance is also improved when simultaneously processing search requests from multiple users.

[1076] An "information communication device" is a device that provides communication history data, such as an email server or chat server.

[1077] "Communication history data" refers to data that includes information such as the content of emails and chat messages, senders and receivers, and dates and times.

[1078] The "standard format" is a data format for converting communication history data in different formats into a unified format.

[1079] The "storage device" is a storage medium such as a database for storing communication history data.

[1080] "Search criteria" are search criteria such as keywords, date ranges, senders and recipients specified by the user.

[1081] The "display means" is an interface for visually presenting search results to the user.

[1082] "Emotion recognition" is a technology that analyzes and recognizes the emotions expressed by users when they enter search criteria.

[1083] "Search result tailoring" refers to changing the priorities of search results based on perceived user sentiment.

[1084] "Changing the display format" means changing the presentation method of search results according to the user's emotions.

[1085] This invention combines a system that acquires history data from various information and communication devices, converts it into a standard format, stores it in a storage device, searches for corresponding history data based on search conditions specified by the user, and visualizes it for the user with an emotion engine that recognizes the user's emotions. This system can adjust search results and change the presentation method based on the user's emotions.

[1086] The server first obtains communication history data from various information and communication devices (e.g., mail servers, chat servers). Specifically, it obtains new emails from the mail server using the IMAP or POP3 protocol, and then obtains messages using the chat tool's API. Specific software libraries called IMAPClient and Requests are used.

[1087] To convert the retrieved data into a standard format, extract the subject, body, sender, recipient, and date and time from email data, and extract the message content, sender, recipient, and timestamp information from chat data. To extract the subject from email data, use email.message_from_string(raw_email)['Subject'], and to extract the message content from chat data, use json.loads(response.text)['messages'].

[1088] Next, the converted data is saved to a storage device (database) and checked for duplicates. To save to the database, an SQL query (e.g., INSERT INTO communications (subject, body, sender, recipient, date) VALUES ...) is executed. A duplicate check is performed using an SQL query (e.g., SELECT COUNT() FROM communications WHERE ...).

[1089] Users access a specialized search interface and enter search criteria, such as search keywords, date ranges, specific senders and recipients, using text boxes, drop-down menus, and calendar widgets. When the user presses a search button, the search criteria are sent to the server.

[1090] Next, the server receives a search request from the user and generates an SQL query based on the search criteria, for example, sql_query = f"SELECT FROM communications WHERE subject LIKE '%{keyword}%' AND date BETWEEN '{start_date}' AND '{end_date}'". It executes this SQL query, retrieves the results, and sorts them by chronological order or relevance.

[1091] Furthermore, the server uses an emotion engine to recognize the user's emotions before formatting the search results. To recognize the user's emotions, it uses an emotion recognition API such as emotion = analyze_emotion(user_input). Based on the recognized emotion, it adjusts the search results. For example, if stress is recognized, it reorders the search results using the method results = prioritize_important_information(results). The presentation method of the search results is also changed accordingly.

[1092] Search results are displayed on the user's device. For emails, the results show the subject, sender, recipient, and date and time, and for chats, a preview of the message is displayed. Users can click on a result to view more information. The history of a specific email or chat can be displayed in a modal window or on a new page, with other related messages displayed as links.

[1093] Specific examples

[1094] 1. User A accesses the system's search interface to investigate past response history regarding "Project Y."

[1095] 2. User A enters "Project Y" as a keyword and sets the date range to the past six months. The emotion engine recognizes from User A's voice and text that he or she is feeling stressed.

[1096] 3. When the search button is pressed, the server searches the database for emails and chat history related to "Project Y," prioritizes the most important information based on the stress level recognized by the emotion engine, and returns it to User A's device.

[1097] 4. User A reviews the list of search results, clicks on the subject of the relevant email to read the full text and extract the key points.

[1098] Prompt Sentence Examples

[1099] I want to investigate the past history of projects related to Project Y. The emotion engine should recognize the user's emotions and prioritize the most important information. Then, display the search results based on color and presentation style.

[1100] In this way, the proposed system takes into account the user's emotions, quickly and efficiently integrates distributed communication histories, and allows users to easily obtain the information they need.

[1101] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1102] Step 1:

[1103] server:

[1104] Input: Connection information and authentication information obtained from various information and communication devices.

[1105] Operation: The server connects to various information and communication devices (e.g., mail servers, chat servers) and acquires communication history data. Specifically, it acquires new emails from the mail server using the IMAP or POP3 protocol, and acquires messages using the chat tool's API.

[1106] Output: Acquired communication history data (e.g., content of emails and chat messages, sender / receiver, date and time, etc.).

[1107] Step 2:

[1108] server:

[1109] Input: Acquired communication history data (e.g., raw email data, chat message data).

[1110] Operation: The server converts the acquired communication history data into a standard format. From email data, it extracts the subject, body, sender, recipient, and date and time, and from chat data, it extracts the message content, sender, recipient, and timestamp information. Specifically, to extract the subject from email data, it uses email.message_from_string(raw_email)['Subject'], and to extract the message content from chat data, it uses json.loads(response.text)['messages'].

[1111] Output: Communication history data converted into a standard format.

[1112] Step 3:

[1113] server:

[1114] Input: Communication history data converted into a standard format.

[1115] Operation: The server converts the communication history data into a standard format, stores it in a storage device (database), and checks for duplicates. Specifically, to store it in the database, it executes the SQL query INSERT INTO communications (subject, body, sender, recipient, date) VALUES ..., and to check for duplicates, it uses the SQL query SELECT COUNT() FROM communications WHERE ....

[1116] Output: Saved communication history data and whether there are any duplicates.

[1117] Step 4:

[1118] User:

[1119] Input: Access the search interface, search criteria such as search keywords, date range, or specific senders / receivers.

[1120] How it works: Users access a dedicated search interface and enter search criteria using text boxes, drop-down menus, and calendar widgets.

[1121] Output: The search criteria that was set.

[1122] Step 5:

[1123] server:

[1124] Input: The search criteria submitted by the user.

[1125] How it works: The server receives a search request from a user and generates an SQL query based on the search criteria. Specifically, it generates an SQL query in the format sql_query = f"SELECT FROM communications WHERE subject LIKE '%{keyword}%' AND date BETWEEN '{start_date}' AND '{end_date}'".

[1126] Output: The generated SQL query.

[1127] Step 6:

[1128] server:

[1129] Input: The generated SQL query.

[1130] How it works: The server executes an SQL query and retrieves search results. The server then executes an SQL query to sort the results in chronological order or relevance order, such as ORDER BY date DESC.

[1131] Output: Sorted search results.

[1132] Step 7:

[1133] server:

[1134] Input: Sorted search results.

[1135] Operation: The server uses the emotion engine to recognize the user's emotions. Specifically, it analyzes the text and voice input of the user's search criteria and calls the emotion recognition API, emotion = analyze_emotion(user_input).

[1136] Output: Perceived user sentiment.

[1137] Step 8:

[1138] server:

[1139] Input: Perceived user sentiment, sorted search results.

[1140] How it works: The server tailors search results based on the perceived emotion. For example, if stress is perceived, it reorders the search results to prioritize more important information using the method results = prioritize_important_information(results).

[1141] Output: Refined search results.

[1142] Step 9:

[1143] server:

[1144] Input: Perceived user sentiment, adjusted search results.

[1145] How it works: The server changes the presentation of search results based on the recognized emotion, for example, presenting vibrant colors for positive emotions and calm colors for negative emotions, using theme-changing methods like if emotion == 'positive': set_vibrant_theme() else: set_calm_theme().

[1146] Output: The format in which the final search results are displayed.

[1147] Step 10:

[1148] Device:

[1149] Input: The format in which the final search results will be displayed.

[1150] What it does: Search results returned by the server are displayed on the user's screen. For emails, the search results show the subject, sender / receiver, and date and time, and for chats, a preview of the message is displayed.

[1151] Output: Search results displayed on the user's screen.

[1152] Step 11:

[1153] User:

[1154] Input: The search result item the user selects on the screen.

[1155] What it does: Users click on a search result item to learn more, or view a specific email or chat history in a modal window or new page, with other related messages available as links.

[1156] Output: A modal window or new page with more information.

[1157] (Application example 2)

[1158] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1159] Conventional food delivery systems only provide information based on order history, support interaction history, and review comments without considering the user's emotions. As a result, when a user is stressed or in a particular emotional state, appropriate information is not provided, resulting in a decrease in user satisfaction.

[1160] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1161] In this invention, the server includes means for acquiring history data from various communication means, means for converting the history data into a unified format, means for storing the data converted into the unified format in a data storage means, means for searching for corresponding history data from the data storage means based on search conditions specified by a user, means for visualizing the searched history data to the user, an emotion recognition engine for analyzing the user's emotions, means for adjusting search results based on the emotions, and means for changing the presentation format based on the emotions, thereby making it possible to provide appropriate information according to the user's emotions.

[1162] "Various communication means" refers to means for exchanging information via multiple different communication protocols and devices.

[1163] "History data" refers to information about past interactions with a user, and primarily includes order history, support interaction history, review comments, and the like.

[1164] A "uniform format" is a standardized data format for consistently handling data acquired in different formats.

[1165] "Data storage means" refers to a means for saving acquired historical data, and typically includes databases and cloud storage.

[1166] "Search conditions" are key information that a user specifies when retrieving specific data, and include keywords, date ranges, specific senders and receivers, and the like.

[1167] The "visualization means" is a means for visually displaying search results to the user, such as a graphical user interface.

[1168] An "emotion recognition engine" is an engine that has the function of analyzing a user's emotions, and is a technology that analyzes emotions from a user's speech or text input.

[1169] The "means for adjusting search results based on emotions" refers to a means for changing the content and display order of search results according to the user's emotions analyzed by the emotion recognition engine.

[1170] The "means for changing the presentation method based on emotions" is a means for adjusting the display method of search results and the UI theme based on the user's emotions.

[1171] The embodiments for carrying out the present invention are as follows.

[1172] Data collection and storage

[1173] The server has a means of acquiring historical data from various communication methods. Specifically, data such as email, chat messages, and order history is acquired using multiple APIs and communication protocols (e.g., IMAP, POP3, HTTP). The acquired historical data is converted into a unified format. For example, the subject, body, sender, sender, and date / time are extracted from email, and the message content, sender, receiver, and timestamp information are extracted from chat messages. The converted data is then saved in an SQLite database, which serves as the data storage method. A check for duplicate data is also performed when saving the data.

[1174] Data Search

[1175] A user accesses a dedicated search interface and enters search criteria. For example, in a food delivery service application, a user can specify keywords and a date range for order history. The server receives the user's search request and generates an SQL query. For example, an SQL statement specifying the keyword "order" and a specific date range is generated. The generated SQL query is executed against the database to retrieve the corresponding history data.

[1176] Sentiment Analysis Engine

[1177] When a user enters search criteria, the emotion recognition engine analyzes the user's emotions. This includes the text the user enters and speech using speech recognition technology. This determines the emotional state as positive, negative, or neutral. The emotion recognition engine uses TextBlob and other natural language processing libraries.

[1178] Tailoring and displaying search results

[1179] The server adjusts search results based on the emotions analyzed by the emotion recognition engine. For example, if the user is feeling stressed, it prioritizes displaying information that is of high importance. It also changes the presentation style based on the emotion, applying a brightly colored design for positive emotions and a more subdued design for negative emotions. The adjusted search results are visualized on the user's device. The results may include, for example, order history details or support interaction history, displayed in chronological order or by relevance.

[1180] Specific examples

[1181] If a user types "my order is late," the server uses an emotion recognition engine to analyze negative emotions. It then adjusts search results, prioritizing important information and visualizing the results in calming colors. Users can then respond quickly based on the displayed results, reducing stress.

[1182] Prompt Sentence Examples

[1183] Use the Food Delivery Emotional Assistant to investigate my pizza order history. I'm stressed because my order is late.

[1184] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1185] Step 1:

[1186] The server acquires historical data from various communication methods. Specifically, it collects new orders, chat messages, and review comments via APIs such as mail servers and chat servers. It receives response data from the API as input and generates a list of the acquired raw data as output.

[1187] Step 2:

[1188] The server converts the acquired historical data into a unified format. Specifically, it extracts the subject, body, sender, recipient, and date and time from email data, and the message content, sender, recipient, and timestamp information from chat messages. It receives a list of raw data as input and generates a list of standardized data as output.

[1189] Step 3:

[1190] The server saves the converted data in a database, which is a data storage means. Specifically, it inserts the data into an SQLite database and also checks for duplicates. It receives a list of standardized data as input and generates the number of records of the data stored in the database as output.

[1191] Step 4:

[1192] A user accesses a dedicated search interface and enters search criteria, such as search keywords and date ranges. The system takes the search criteria specified by the user as input and generates a query based on the search criteria as output.

[1193] Step 5:

[1194] The server generates an SQL query based on the search criteria it receives. Specifically, it creates an SQL statement that includes search keywords and date ranges. It takes the search criteria specified by the user as input and generates an executable SQL statement as output.

[1195] Step 6:

[1196] The server executes the generated SQL query against the database to search for the corresponding historical data. Specifically, it retrieves the relevant records from the database and sorts them in chronological order or by relevance. It receives the SQL statement as input and generates a list of search results as output.

[1197] Step 7:

[1198] The server uses an emotion recognition engine to analyze the user's emotions. Specifically, it analyzes the text and voice input by the user and determines whether the emotion is positive, negative, or neutral. It receives the user's text and voice data as input and generates the type of emotion as output.

[1199] Step 8:

[1200] The server adjusts search results based on the user's emotions. Specifically, it sorts the search results to prioritize the most important information. As input, it receives the emotion type and a list of search results, and as output, it generates a list of adjusted search results.

[1201] Step 9:

[1202] The server displays the adjusted search results in an appropriate presentation style, specifically, applying a vibrant design for positive emotions and a subdued design for negative emotions. The server receives the adjusted search results list as input and generates a search result screen to be displayed on the user's device as output.

[1203] Step 10:

[1204] The user can interact with the displayed search results and view more information. For example, they can click on a specific order history or support interaction history to view its contents. The system takes the displayed search results as input and displays a modal window or a new page with more information as output.

[1205] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1206] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1207] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1208] [Fourth embodiment]

[1209] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1210] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1211] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1212] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1213] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1214] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1215] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1216] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1217] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1218] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1219] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1220] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1221] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1222] This invention provides a specific method for implementing a system that acquires historical data from various communication tools, converts it into a unified format, stores it in a database, searches for corresponding historical data based on search conditions specified by the user, and visualizes it for the user.

[1223] Data collection

[1224] server:

[1225] 1. The server periodically retrieves historical data from each communication tool (e.g., mail server, chat server). This includes retrieving new emails from the mail server using the IMAP or POP3 protocol, and retrieving messages using the chat tool's API.

[1226] 2. A parser is used to convert the retrieved data into a unified format. For example, for email data, information such as the subject, body, sender, recipient, and date and time is retrieved, and for chat data, message content, sender, recipient, and timestamp information is retrieved.

[1227] 3. The converted data is saved in the database. This data is checked for duplicates and saved in the database in a unified format.

[1228] Data Search

[1229] User:

[1230] 1. The user accesses a dedicated search interface and enters search criteria such as search keywords, a date range, or specific senders / receivers. The interface is presented using text boxes, drop-down menus, and calendar widgets.

[1231] 2. Set the search conditions and press the search button.

[1232] server:

[1233] 1. The server receives a search request from the user and generates a SQL query based on the criteria. For example, if you specify the keyword "Project X" and a specific date range, it generates a SQL statement to search for historical data that matches those criteria.

[1234] 2. The generated SQL query is executed against the database to retrieve search results, which are then sorted by chronological order and relevance.

[1235] Display and Operation

[1236] Device:

[1237] 1. The search results returned from the server are displayed on the user's screen. For emails, the search results include the subject, sender, recipient, and date and time, and for chats, a preview of the message is displayed.

[1238] 2. Users can click on individual search results to view more information. For example, clicking the Details button will open a modal window or new page with the full text of the email or chat history.

[1239] Specific examples

[1240] scenario:

[1241] 1. User A wants to investigate past response history regarding "Project Y" and accesses the system's search interface.

[1242] 2. User A enters "Project Y" as a keyword and sets the date range to the past 6 months.

[1243] 3. When the search button is pressed, the server searches the database for emails and chat history related to "Project Y," formats it, and returns it to User A's device.

[1244] 4. User A reviews the list of search results, clicks on the subject of the relevant email to read the full text and extract the key points.

[1245] In this way, the proposed system quickly and efficiently integrates distributed communication histories, allowing users to easily obtain the information they need.

[1246] The processing flow will be explained below.

[1247] Step 1: Get the data

[1248] server:

[1249] The server is configured to periodically retrieve new messages and historical data using the APIs or protocols of various communication tools (e.g., IMAP, POP3).

[1250] To retrieve new mail, the IMAP protocol is used to retrieve new mail from each user's inbox.

[1251] Call the chat tool's public API to get new messages from that tool.

[1252] Step 2: Transform the data

[1253] server:

[1254] The acquired data is temporarily saved and converted into a unified format (e.g., JSON format).

[1255] In the case of email data, information such as the subject, body, sender / receiver, date and time is extracted.

[1256] For chat data, message content, sender, recipient, and timestamp are extracted.

[1257] Step 3: Save your data

[1258] server:

[1259] The converted data is checked for duplicates and saved in the database.

[1260] Save new data to the database using INSERT operations and, if necessary, update operations.

[1261] Index the database to speed up later searches.

[1262] Step 4: Enter search criteria

[1263] User:

[1264] Users access a search interface and enter criteria such as search keywords, date ranges, and specific senders and recipients.

[1265] The interface allows you to enter search keywords into the text box and set a date range using the calendar widget.

[1266] Click the search button.

[1267] Step 5: Generating a search query

[1268] server:

[1269] It receives search requests from users and generates SQL queries based on the conditions.

[1270] For example, dynamically generate a SQL statement specifying keywords and a date range related to "Project X."

[1271] The generated query is sent to the database.

[1272] Step 6: Database Search

[1273] server:

[1274] Run the generated SQL queries against the database to find relevant historical data.

[1275] Search results are sorted chronologically and by relevance.

[1276] Step 7: Formatting the search results

[1277] server:

[1278] Format the search results and create a response to return to the user.

[1279] Generates JSON and HTML responses and includes additional information for each search result (e.g., email previews, chat message excerpts).

[1280] Step 8: Viewing search results

[1281] Device:

[1282] The search results returned from the server are displayed on the user's screen.

[1283] Results can be sorted chronologically or by relevance, and for emails, the subject, sender, sender, and date and time are displayed.

[1284] Step 9: View details and take action

[1285] User:

[1286] Users click on search result items to see more details.

[1287] View a specific email or chat history in a modal window or new page, with other related messages displayed as links.

[1288] Example 1

[1289] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1290] Conventional information retrieval systems face the problem of difficulty in centrally managing historical information from various information transmission tools, making it difficult to quickly search for and display specific information. Furthermore, performance often declines when processing search requests from multiple users in parallel. Another issue is the lack of a means to effectively sort and display search results.

[1291] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1292] In this invention, the server includes means for acquiring history information from various information transmission tools, means for converting the history information into a unified format, means for storing the information converted into the unified format in a data warehouse, means for searching the data warehouse for corresponding history information based on search conditions specified by a user, means for visualizing the searched history information to the user, means for periodically acquiring history information, means for checking for duplication and storing the information, means for accessing a dedicated search interface, means for inputting search keywords, date ranges, and specific senders and receivers, means for generating a query language based on the conditions, and means for sorting the acquired data in chronological order or by relevance. This makes it possible to efficiently collect, manage, and search history information from distributed information transmission tools and quickly respond to search requests from multiple users.

[1293] "Information communication tools" refers to communication methods such as email, chat, and messaging applications.

[1294] "History information" refers to messages and data sent and received through various information transmission tools, as well as related metadata.

[1295] "Unified format" refers to a format that converts data obtained from different information delivery tools into a consistent format.

[1296] "Data warehouse" refers to a database system designed to store and manage acquired historical information and enable efficient search and extraction.

[1297] "Search conditions" refer to conditions such as keywords, date ranges, senders, and recipients that a user specifies when searching for specific information.

[1298] "Duplicate check" refers to a confirmation process to prevent the same history information from being saved twice.

[1299] "Search interface" refers to a user interface that allows a user to input search criteria and view search results.

[1300] "Query language" refers to a query language (e.g., SQL) used to retrieve data from a database under specific conditions.

[1301] "Chronological order" refers to the order in which data is arranged in the order in which it occurred.

[1302] "Sort by relevance" refers to the order in which data is arranged in descending order of relevance to the search criteria.

[1303] This invention provides a specific method for implementing a system that acquires historical information from various information transmission tools, converts it into a unified format, stores it in a data warehouse, searches for corresponding historical information based on search conditions specified by the user, and visualizes it for the user.

[1304] First, the server periodically collects historical information from multiple information transmission tools. It retrieves emails from the mail server using the IMAP or POP3 protocol, and retrieves messages from the chat tool via API. It uses a cron job to check for new data every hour. The collected data is temporarily stored in local storage.

[1305] The server then runs a parser to convert the collected history information into a unified format. For example, a Python script is used to extract information such as the subject, body, sender, recipient, and date and time from emails and convert them into JSON objects. Similarly, chat messages are converted into a unified format, with information such as the message content, sender, recipient, and timestamp.

[1306] The server checks for duplicates of the converted data before storing it in a data warehouse (e.g., MySQL, PostgreSQL). Message IDs and timestamps are used to check for duplicates. Data stored in the data warehouse is designed to be efficiently searchable and retrievalable.

[1307] Users access a dedicated search interface and enter search criteria (e.g., search keywords, date ranges, specific senders / receivers). The search interface is provided using text boxes, drop-down menus, and calendar widgets. When a search button is pressed, the server receives the search request and generates an SQL query based on the received criteria.

[1308] For example, if the conditions "Project Y" and "Last 6 months" are specified, the server generates an SQL query such as "SELECT FROM communication_logs WHERE body LIKE '%ProjectY%' AND timestamp BETWEEN 'Start Date' AND 'End Date'". The server then queries the database and retrieves the search results, which are sorted chronologically and by relevance.

[1309] The device displays the search results returned by the server on the user's screen. For emails, the subject, sender, recipient, and date and time are displayed, and for chats, a preview of the message is displayed. The user can click on individual search results to view more information.

[1310] Specific examples

[1311] scenario:

[1312] 1. User A wants to investigate past response history regarding "Project Y" and accesses the system's search interface.

[1313] 2. User A enters "Project Y" as a keyword and sets the date range to the past 6 months.

[1314] 3. When the search button is pressed, the server searches the data warehouse for emails and chat history related to "Project Y," formats the data, and returns it to User A's device.

[1315] 4. User A reviews the list of search results, clicks on the subject of the relevant email to read the full text and extract the key points.

[1316] Example prompts for generative AI models

[1317] "Please explain the algorithm for a system that converts historical information obtained from various information transmission tools into a unified format and stores it in a data warehouse. This includes the ability for users to search for specific keywords or date ranges."

[1318] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1319] Step 1:

[1320] The server obtains history information from the information transmission tool. For example, it obtains messages from the mail server using the IMAP or POP3 protocol, and from the chat tool via an API. Input data includes authentication information for the mail server and the API key for the chat tool. Based on this, new emails and messages are temporarily saved in local storage.

[1321] Step 2:

[1322] The server reads the history information from the temporary storage folder and converts it into a unified format. The input data is history information files in JSON and CSV format. Specifically, a Python script is executed to extract the email subject, body, sender / receiver, date and time, etc., and convert them into a unified JSON object. The output data is a unified JSON object containing the subject, body, sender, recipient, and timestamp information.

[1323] Step 3:

[1324] The server checks for duplicates in the converted data before storing it in the data warehouse. The input data is a unified JSON object. An SQL statement such as "INSERT INTO communication_logs (subject, body, sender, receiver, timestamp) VALUES (...) ON DUPLICATE KEY UPDATE ..." is executed against the database. The output is accurate historical information stored in the data warehouse.

[1325] Step 4:

[1326] Users access a dedicated search interface and enter search criteria. The input data can include search keywords, a date range, and specific senders and recipients. Based on this, the search interface receives the entered criteria and displays them on the screen. The output is the received search criteria.

[1327] Step 5:

[1328] The server receives a search request from the user and generates an SQL query based on the conditions. For example, if "Project Y" and "Last 6 months" are specified, it generates an SQL query such as "SELECT FROM communication_logs WHERE body LIKE '%ProjectY%' AND timestamp BETWEEN 'Start date' AND 'End date'". The input data is the search conditions, and the output data is the generated SQL query.

[1329] Step 6:

[1330] The server executes the generated SQL query and retrieves the search results from the data warehouse. This involves executing the SQL query against the database and retrieving the relevant data. The input data is the SQL query, and the output data is the dataset of the search results.

[1331] Step 7:

[1332] The server sorts and formats the search results in chronological order or relevance order. For example, it sorts the list of search results by timestamp. The input data is the dataset of search results, and the output data is the sorted dataset.

[1333] Step 8:

[1334] The device displays the search results returned by the server on the user's screen: for emails, the subject, sender, recipient, and date and time are displayed, and for chats, a preview of the message is displayed. The input data is a sorted dataset, and the output is the search results displayed in the user's browser.

[1335] Step 9:

[1336] The user selects the information they need from the displayed list of search results and checks the details. For example, they click on the subject of a specific email to read the full text and extract important points. The input data is the displayed search results, and the output data is the email content and chat history displayed in the detailed view.

[1337] (Application example 1)

[1338] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1339] Currently, security service inquiries and trouble report history management is distributed across multiple communication tools (email, chat, etc.), and there is an inadequate system for integrated management and search. As a result, information sharing between end users and support teams is delayed, making it difficult to respond quickly. Furthermore, when searching for related data, users often cannot specify appropriate search criteria, which often hinders efficient information retrieval. These issues need to be resolved.

[1340] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1341] In this invention, the server includes means for acquiring history data from various communication tools, means for converting the history data into a unified format, means for storing the data converted into the unified format in a database, means for searching the database for corresponding history data based on search criteria specified by a user, means for visualizing the searched history data to the user, means for acquiring history data using data collection software and interface software, converting the data into a unified format, and storing the data, means for performing search and visualization through an application compatible with the user's smart device, and means for recommending search criteria and automatically generating prompt sentences using a generative AI model. This allows for centralized management of distributed communication histories between end users and support teams, enabling rapid and appropriate information search and sharing.

[1342] "Various communication tools" is a general term for communication methods using different media, such as email, chat applications, and messaging platforms.

[1343] "Historical Data" means records of messages, emails, and other communications exchanged through various communication tools.

[1344] A "uniform format" is a standardized data format for converting different types of historical data into a consistent format.

[1345] "Database" means a centralized data management system for efficiently storing, searching, and managing historical data that has been converted into a unified format.

[1346] "Search criteria" are criteria such as keywords, date ranges, senders, and recipients that a user specifies to identify and search for specific historical data.

[1347] "Visualization" is the process of displaying retrieved historical data in a way that is easy for users to understand.

[1348] "Data collection software" is a program for automatically collecting historical data from various communication tools.

[1349] "Interface software" refers to the operating screens and applications that allow users to interact with the data collection software and database.

[1350] "Smart devices" are highly functional portable devices such as smartphones, tablets, smart glasses, and head-mounted displays.

[1351] A "generative AI model" is a computational model that uses artificial intelligence techniques to automatically perform specific tasks.

[1352] "Search criterion recommendation" is a function in which the generative AI model automatically suggests appropriate search criteria so that users can specify them.

[1353] "Automatic prompt generation" is the process of using a generative AI model to automatically generate commands that users use when searching or performing operations.

[1354] The present invention provides a system that uses a user's smart device to centrally manage, search, and visualize historical data of inquiries and trouble reports for security services. Specific embodiments of this system will be described below.

[1355] Specific methods for collecting data

[1356] server:

[1357] The server periodically retrieves historical data from various communication tools. For example, it retrieves emails from a mail server using the IMAP or POP3 protocol, and retrieves messages from chat tools using an API. A parser is used to convert the retrieved data into a unified format. In the case of email data, information such as the subject, body, sender / receiver, and date / time is retrieved, and in the case of chat data, information on the message content, sender, recipient, and timestamp is retrieved. The converted data is stored in a database. This database checks for duplicates and manages the data in a unified format.

[1358] Specific methods for data search

[1359] User:

[1360] Users access a dedicated search interface and enter search criteria such as search keywords, date ranges, and specific senders and recipients. The interface is provided through a smart device application. When the user presses the search button, the server receives the search request and generates an SQL query based on the criteria. The generated SQL query is used to search corresponding historical data from the database and retrieve search results. The search results are then sorted in chronological order or by relevance.

[1361] Specific display and operation methods

[1362] Device:

[1363] The application installed on the user's smart device displays the search results returned from the server on the user's screen. For emails, the search results display the subject, sender, recipient, and date and time, while for chats, a preview of the message is displayed. The user can click on individual items in the search results to view more information. Pressing the details button displays the full text of the relevant email or chat history.

[1364] Utilizing generative AI models and prompts

[1365] Generative AI models:

[1366] The server uses a generative AI model to provide a function that recommends search terms entered by the user. For example, if a user enters an ambiguous keyword, the server automatically suggests specific search terms related to that keyword. Furthermore, the server also automatically generates prompt sentences to help users perform searches efficiently.

[1367] Examples:

[1368] User A wants to investigate past correspondence history regarding "Project Y" and accesses the system's search interface. User A enters "Project Y" as a keyword and sets the date range to the past six months. When he presses the search button, the server searches the database for emails and chat history related to "Project Y," formats it, and returns it to User A's device. User A checks the list of search results displayed, clicks on the subject of a related email to read the full text, and extracts important points.

[1369] Example prompt sentence:

[1370] A user wants to search for past security-related incident history using specific keywords. When the user enters "Keyword: security incident" and "Date range: January 1, 2023 to December 31, 2023", generate Python code to retrieve the above historical data and display relevant results.

[1371] In this way, the proposed system quickly and efficiently integrates distributed communication histories, allowing users to easily obtain the information they need.

[1372] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1373] Step 1:

[1374] The server acquires historical data from various communication tools (e.g., mail servers, chat APIs). Specifically, it acquires email data using the IMAP or POP3 protocol, and message data using the chat tool's API. The input is raw data from each communication tool, and the output is the acquired raw data.

[1375] Step 2:

[1376] The server converts the acquired raw data into a unified format. For example, email data is broken down into items such as subject, body, sender, sender, and date and time, while chat data is broken down into message content, sender, receiver, and timestamp information. The input is raw data, and the output is data converted into a unified format.

[1377] Step 3:

[1378] The server saves the data converted into a unified format in the database. At the same time, it checks for duplicates and removes items that overlap with existing data. The input is unified format data, and the output is the data saved in the database.

[1379] Step 4:

[1380] A user accesses a dedicated search interface installed on a smart device and inputs search criteria such as search keywords, date range, sender / receiver, etc. The input is the search criteria specified by the user, and the output is a search request sent to the search interface.

[1381] Step 5:

[1382] The server receives a search request from a user, generates an SQL query based on the conditions, and searches the corresponding historical data from the database. The input is the search request from the user, and the output is the historical data in the database that matches the search conditions.

[1383] Step 6:

[1384] The server sorts the search results in chronological order or by relevance. The input is the searched history data, and the output is the sorted search results.

[1385] Step 7:

[1386] The server uses a generative AI model to automatically generate search query recommendations and prompts for users. The input is the initial search query and data in the database, and the output is the recommended search query and prompt.

[1387] Step 8:

[1388] The terminal displays the search results returned from the server. For emails, the search results show the subject, sender, recipient, and date and time, and for chats, a preview of the message is displayed. The input is the search results from the server, and the output is the search results in list format that is displayed on the user's screen.

[1389] Step 9:

[1390] Users can click on individual items from the displayed search results to check detailed information. Pressing the details button displays the full text of the relevant email or chat history. The input is the user's click, and the output is a popup or separate screen displaying detailed information.

[1391] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1392] This invention combines a system that acquires history data from various communication tools, converts it into a unified format, stores it in a database, searches for corresponding history data based on search conditions specified by the user, and visualizes it for the user with an emotion engine that recognizes the user's emotions. This system can adjust search results and change presentation methods based on the user's emotions.

[1393] Data collection

[1394] server:

[1395] 1. The server obtains historical data from various communication tools (e.g., mail servers, chat servers). Examples include obtaining new emails from a mail server using the IMAP or POP3 protocol, or obtaining messages using a chat tool's API.

[1396] 2. To convert the acquired data into a unified format, the subject, body, sender, sender, and date / time are extracted from email data, and the message content, sender, receiver, and timestamp information are extracted from chat data.

[1397] 3. Save the converted data in the database and check for duplicates.

[1398] Data Search

[1399] User:

[1400] 1. The user accesses a dedicated search interface and enters search criteria such as search keywords, a date range, or specific senders / receivers. The input interface uses text boxes, drop-down menus, and calendar widgets.

[1401] 2. Set the search conditions and press the search button.

[1402] server:

[1403] 1. The server receives a search request from a user and generates a SQL query, for example, specifying the keyword "Project X" and a specific date range.

[1404] 2. Run the generated SQL query against the database to retrieve search results, sorted chronologically and / or by relevance.

[1405] Emotion engine processing

[1406] server:

[1407] 1. Before formatting the search results, we use an emotion engine to recognize the user's emotions by analyzing the user's speech and input text when entering search criteria.

[1408] 2. Tailor search results based on perceived emotions, for example, prioritizing more important information if a user is feeling stressed.

[1409] 3. The presentation of search results will also change based on emotion, for example, results will be displayed in vibrant colors for positive emotions and muted colors for negative emotions.

[1410] Display and Operation

[1411] Device:

[1412] 1. The search results returned from the server are displayed on the user's screen. For emails, the results show the subject, sender, recipient, and date and time, and for chats, a preview of the message is displayed.

[1413] 2. Users can click on search results to learn more, displaying a specific email or chat history in a modal window or new page, with other related messages available as links.

[1414] Specific examples

[1415] scenario:

[1416] 1. User A accesses the system's search interface to investigate past response history regarding "Project Y."

[1417] 2. User A enters "Project Y" as a keyword and sets the date range to the past six months. The emotion engine recognizes from User A's voice and text that he or she is feeling stressed.

[1418] 3. When the search button is pressed, the server searches the database for emails and chat history related to "Project Y," prioritizes the most important information based on the stress level recognized by the emotion engine, and returns it to User A's device.

[1419] 4. User A reviews the list of search results, clicks on the subject of the relevant email to read the full text and extract the key points.

[1420] In this way, the proposed system takes into account the user's emotions, quickly and efficiently integrates distributed communication histories, and allows users to easily obtain the information they need.

[1421] The processing flow will be explained below.

[1422] Step 1: Get the data

[1423] server:

[1424] The server is configured to periodically retrieve new messages and historical data using the APIs or protocols of various communication tools (e.g., IMAP, POP3).

[1425] To retrieve new mail, the IMAP protocol is used to retrieve new mail from each user's inbox.

[1426] Call the chat tool's public API to get new messages from that tool.

[1427] Step 2: Transform the data

[1428] server:

[1429] The acquired data is temporarily saved and converted into a unified format (e.g., JSON format).

[1430] In the case of email data, information such as the subject, body, sender / receiver, date and time is extracted.

[1431] For chat data, message content, sender, recipient, and timestamp are extracted.

[1432] Step 3: Save your data

[1433] server:

[1434] The converted data is checked for duplicates and saved in the database.

[1435] Save new data to the database using INSERT operations and, if necessary, update operations.

[1436] Index the database to speed up later searches.

[1437] Step 4: Enter search criteria

[1438] User:

[1439] Users access a search interface and enter criteria such as search keywords, date ranges, and specific senders and recipients.

[1440] The interface allows you to enter search keywords into the text box and set a date range using the calendar widget.

[1441] Click the search button.

[1442] Step 5: Recognize emotions

[1443] server:

[1444] The emotion engine recognizes the user's emotions through text analysis of the user's input (keywords and comments) or, in the case of voice input, through voice analysis.

[1445] Data about the recognized emotions is temporarily stored for use in adjusting search queries and result display.

[1446] Step 6: Generating a search query

[1447] server:

[1448] It receives a search request from the user and generates an SQL query taking into account the results of the emotion engine.

[1449] For example, dynamically generate SQL statements specifying keywords related to "Project X" and a specific date range, and prioritize the importance of the results if the user's sentiment is negative.

[1450] Step 7: Database Search

[1451] server:

[1452] Run the generated SQL queries against the database to find relevant historical data.

[1453] Search results are sorted chronologically and by relevance based on user sentiment.

[1454] Step 8: Formatting the search results

[1455] server:

[1456] Format the search results and create a response to return to the user.

[1457] Generates JSON and HTML responses and includes additional information for each search result (e.g., email previews, chat message excerpts).

[1458] Based on the results of the emotion engine, the way search results are displayed (color tone, layout, etc.) is adjusted.

[1459] Step 9: Viewing search results

[1460] Device:

[1461] The search results returned from the server are displayed on the user's screen.

[1462] Results can be sorted chronologically or by relevance, and for emails, the subject, sender, sender, and date and time are displayed.

[1463] Step 10: View details and operate

[1464] User:

[1465] Users click on search result items to see more details.

[1466] View a specific email or chat history in a modal window or new page, with other related messages displayed as links.

[1467] Example 2

[1468] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1469] Conventional communication history search systems make it difficult for users to efficiently retrieve the information they need from the vast number of emails and chat histories. Furthermore, they are unable to prioritize information display based on the user's emotions or change the display format, so there is a need for improved user experience. Furthermore, processing search requests from multiple users simultaneously places a heavy load on the system, resulting in a decrease in system performance.

[1470] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1471] In this invention, the server includes means for acquiring communication history data from various information communication devices, means for converting the communication history data into a standard format, means for storing the data converted into the standard format in a storage device, means for searching the storage device for corresponding communication history data based on search criteria specified by a user, means for displaying the searched communication history data to the user, means for recognizing the user's emotion, means for adjusting the search results based on the recognized emotion, and means for changing the display format of the search results based on the recognized emotion. This enables prioritized display of information and change of the display format according to the user's emotion, allowing necessary information to be obtained quickly and efficiently. System performance is also improved when simultaneously processing search requests from multiple users.

[1472] An "information communication device" is a device that provides communication history data, such as an email server or chat server.

[1473] "Communication history data" refers to data that includes information such as the content of emails and chat messages, senders and receivers, and dates and times.

[1474] The "standard format" is a data format for converting communication history data in different formats into a unified format.

[1475] The "storage device" is a storage medium such as a database for storing communication history data.

[1476] "Search criteria" are search criteria such as keywords, date ranges, senders and recipients specified by the user.

[1477] The "display means" is an interface for visually presenting search results to the user.

[1478] "Emotion recognition" is a technology that analyzes and recognizes the emotions expressed by users when they enter search criteria.

[1479] "Search result tailoring" refers to changing the priorities of search results based on perceived user sentiment.

[1480] "Changing the display format" means changing the presentation method of search results according to the user's emotions.

[1481] This invention combines a system that acquires history data from various information and communication devices, converts it into a standard format, stores it in a storage device, searches for corresponding history data based on search conditions specified by the user, and visualizes it for the user with an emotion engine that recognizes the user's emotions. This system can adjust search results and change the presentation method based on the user's emotions.

[1482] The server first obtains communication history data from various information and communication devices (e.g., mail servers, chat servers). Specifically, it obtains new emails from the mail server using the IMAP or POP3 protocol, and then obtains messages using the chat tool's API. Specific software libraries called IMAPClient and Requests are used.

[1483] To convert the retrieved data into a standard format, extract the subject, body, sender, recipient, and date and time from email data, and extract the message content, sender, recipient, and timestamp information from chat data. To extract the subject from email data, use email.message_from_string(raw_email)['Subject'], and to extract the message content from chat data, use json.loads(response.text)['messages'].

[1484] Next, the converted data is saved to a storage device (database) and checked for duplicates. To save to the database, an SQL query (e.g., INSERT INTO communications (subject, body, sender, recipient, date) VALUES ...) is executed. A duplicate check is performed using an SQL query (e.g., SELECT COUNT() FROM communications WHERE ...).

[1485] Users access a specialized search interface and enter search criteria, such as search keywords, date ranges, specific senders and recipients, using text boxes, drop-down menus, and calendar widgets. When the user presses a search button, the search criteria are sent to the server.

[1486] Next, the server receives a search request from the user and generates an SQL query based on the search criteria, for example, sql_query = f"SELECT FROM communications WHERE subject LIKE '%{keyword}%' AND date BETWEEN '{start_date}' AND '{end_date}'". It executes this SQL query, retrieves the results, and sorts them by chronological order or relevance.

[1487] Furthermore, the server uses an emotion engine to recognize the user's emotions before formatting the search results. To recognize the user's emotions, it uses an emotion recognition API such as emotion = analyze_emotion(user_input). Based on the recognized emotion, it adjusts the search results. For example, if stress is recognized, it reorders the search results using the method results = prioritize_important_information(results). The presentation method of the search results is also changed accordingly.

[1488] Search results are displayed on the user's device. For emails, the results show the subject, sender, recipient, and date and time, and for chats, a preview of the message is displayed. Users can click on a result to view more information. The history of a specific email or chat can be displayed in a modal window or on a new page, with other related messages displayed as links.

[1489] Specific examples

[1490] 1. User A accesses the system's search interface to investigate past response history regarding "Project Y."

[1491] 2. User A enters "Project Y" as a keyword and sets the date range to the past six months. The emotion engine recognizes from User A's voice and text that he or she is feeling stressed.

[1492] 3. When the search button is pressed, the server searches the database for emails and chat history related to "Project Y," prioritizes the most important information based on the stress level recognized by the emotion engine, and returns it to User A's device.

[1493] 4. User A reviews the list of search results, clicks on the subject of the relevant email to read the full text and extract the key points.

[1494] Prompt Sentence Examples

[1495] I want to investigate the past history of projects related to Project Y. The emotion engine should recognize the user's emotions and prioritize the most important information. Then, display the search results based on color and presentation style.

[1496] In this way, the proposed system takes into account the user's emotions, quickly and efficiently integrates distributed communication histories, and allows users to easily obtain the information they need.

[1497] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1498] Step 1:

[1499] server:

[1500] Input: Connection information and authentication information obtained from various information and communication devices.

[1501] Operation: The server connects to various information and communication devices (e.g., mail servers, chat servers) and acquires communication history data. Specifically, it acquires new emails from the mail server using the IMAP or POP3 protocol, and acquires messages using the chat tool's API.

[1502] Output: Acquired communication history data (e.g., content of emails and chat messages, sender / receiver, date and time, etc.).

[1503] Step 2:

[1504] server:

[1505] Input: Acquired communication history data (e.g., raw email data, chat message data).

[1506] Operation: The server converts the acquired communication history data into a standard format. From email data, it extracts the subject, body, sender, recipient, and date and time, and from chat data, it extracts the message content, sender, recipient, and timestamp information. Specifically, to extract the subject from email data, it uses email.message_from_string(raw_email)['Subject'], and to extract the message content from chat data, it uses json.loads(response.text)['messages'].

[1507] Output: Communication history data converted into a standard format.

[1508] Step 3:

[1509] server:

[1510] Input: Communication history data converted into a standard format.

[1511] Operation: The server converts the communication history data into a standard format, stores it in a storage device (database), and checks for duplicates. Specifically, to store it in the database, it executes the SQL query INSERT INTO communications (subject, body, sender, recipient, date) VALUES ..., and to check for duplicates, it uses the SQL query SELECT COUNT() FROM communications WHERE ....

[1512] Output: Saved communication history data and whether there are any duplicates.

[1513] Step 4:

[1514] User:

[1515] Input: Access the search interface, search criteria such as search keywords, date range, or specific senders / receivers.

[1516] How it works: Users access a dedicated search interface and enter search criteria using text boxes, drop-down menus, and calendar widgets.

[1517] Output: The search criteria that was set.

[1518] Step 5:

[1519] server:

[1520] Input: The search criteria submitted by the user.

[1521] How it works: The server receives a search request from a user and generates an SQL query based on the search criteria. Specifically, it generates an SQL query in the format sql_query = f"SELECT FROM communications WHERE subject LIKE '%{keyword}%' AND date BETWEEN '{start_date}' AND '{end_date}'".

[1522] Output: The generated SQL query.

[1523] Step 6:

[1524] server:

[1525] Input: The generated SQL query.

[1526] How it works: The server executes an SQL query and retrieves search results. The server then executes an SQL query to sort the results in chronological order or relevance order, such as ORDER BY date DESC.

[1527] Output: Sorted search results.

[1528] Step 7:

[1529] server:

[1530] Input: Sorted search results.

[1531] Operation: The server uses the emotion engine to recognize the user's emotions. Specifically, it analyzes the text and voice input of the user's search criteria and calls the emotion recognition API, emotion = analyze_emotion(user_input).

[1532] Output: Perceived user sentiment.

[1533] Step 8:

[1534] server:

[1535] Input: Perceived user sentiment, sorted search results.

[1536] How it works: The server tailors search results based on the perceived emotion. For example, if stress is perceived, it reorders the search results to prioritize more important information using the method results = prioritize_important_information(results).

[1537] Output: Refined search results.

[1538] Step 9:

[1539] server:

[1540] Input: Perceived user sentiment, adjusted search results.

[1541] How it works: The server changes the presentation of search results based on the recognized emotion, for example, presenting vibrant colors for positive emotions and calm colors for negative emotions, using theme-changing methods like if emotion == 'positive': set_vibrant_theme() else: set_calm_theme().

[1542] Output: The format in which the final search results are displayed.

[1543] Step 10:

[1544] Device:

[1545] Input: The format in which the final search results will be displayed.

[1546] What it does: Search results returned by the server are displayed on the user's screen. For emails, the search results show the subject, sender / receiver, and date and time, and for chats, a preview of the message is displayed.

[1547] Output: Search results displayed on the user's screen.

[1548] Step 11:

[1549] User:

[1550] Input: The search result item the user selects on the screen.

[1551] What it does: Users click on a search result item to learn more, or view a specific email or chat history in a modal window or new page, with other related messages available as links.

[1552] Output: A modal window or new page with more information.

[1553] (Application example 2)

[1554] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1555] Conventional food delivery systems only provide information based on order history, support interaction history, and review comments without considering the user's emotions. As a result, when a user is stressed or in a particular emotional state, appropriate information is not provided, resulting in a decrease in user satisfaction.

[1556] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1557] In this invention, the server includes means for acquiring history data from various communication means, means for converting the history data into a unified format, means for storing the data converted into the unified format in a data storage means, means for searching for corresponding history data from the data storage means based on search conditions specified by a user, means for visualizing the searched history data to the user, an emotion recognition engine for analyzing the user's emotions, means for adjusting search results based on the emotions, and means for changing the presentation format based on the emotions, thereby making it possible to provide appropriate information according to the user's emotions.

[1558] "Various communication means" refers to means for exchanging information via multiple different communication protocols and devices.

[1559] "History data" refers to information about past interactions with a user, and primarily includes order history, support interaction history, review comments, and the like.

[1560] A "uniform format" is a standardized data format for consistently handling data acquired in different formats.

[1561] "Data storage means" refers to a means for saving acquired historical data, and typically includes databases and cloud storage.

[1562] "Search conditions" are key information that a user specifies when retrieving specific data, and include keywords, date ranges, specific senders and receivers, and the like.

[1563] The "visualization means" is a means for visually displaying search results to the user, such as a graphical user interface.

[1564] An "emotion recognition engine" is an engine that has the function of analyzing a user's emotions, and is a technology that analyzes emotions from a user's speech or text input.

[1565] The "means for adjusting search results based on emotions" refers to a means for changing the content and display order of search results according to the user's emotions analyzed by the emotion recognition engine.

[1566] The "means for changing the presentation method based on emotions" is a means for adjusting the display method of search results and the UI theme based on the user's emotions.

[1567] The embodiments for carrying out the present invention are as follows.

[1568] Data collection and storage

[1569] The server has a means of acquiring historical data from various communication methods. Specifically, data such as email, chat messages, and order history is acquired using multiple APIs and communication protocols (e.g., IMAP, POP3, HTTP). The acquired historical data is converted into a unified format. For example, the subject, body, sender, sender, and date / time are extracted from email, and the message content, sender, receiver, and timestamp information are extracted from chat messages. The converted data is then saved in an SQLite database, which serves as the data storage method. A check for duplicate data is also performed when saving the data.

[1570] Data Search

[1571] A user accesses a dedicated search interface and enters search criteria. For example, in a food delivery service application, a user can specify keywords and a date range for order history. The server receives the user's search request and generates an SQL query. For example, an SQL statement specifying the keyword "order" and a specific date range is generated. The generated SQL query is executed against the database to retrieve the corresponding history data.

[1572] Sentiment Analysis Engine

[1573] When a user enters search criteria, the emotion recognition engine analyzes the user's emotions. This includes the text the user enters and speech using speech recognition technology. This determines the emotional state as positive, negative, or neutral. The emotion recognition engine uses TextBlob and other natural language processing libraries.

[1574] Tailoring and displaying search results

[1575] The server adjusts search results based on the emotions analyzed by the emotion recognition engine. For example, if the user is feeling stressed, it prioritizes displaying information that is of high importance. It also changes the presentation style based on the emotion, applying a brightly colored design for positive emotions and a more subdued design for negative emotions. The adjusted search results are visualized on the user's device. The results may include, for example, order history details or support interaction history, displayed in chronological order or by relevance.

[1576] Specific examples

[1577] If a user types "my order is late," the server uses an emotion recognition engine to analyze negative emotions. It then adjusts search results, prioritizing important information and visualizing the results in calming colors. Users can then respond quickly based on the displayed results, reducing stress.

[1578] Prompt Sentence Examples

[1579] Use the Food Delivery Emotional Assistant to investigate my pizza order history. I'm stressed because my order is late.

[1580] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1581] Step 1:

[1582] The server acquires historical data from various communication methods. Specifically, it collects new orders, chat messages, and review comments via APIs such as mail servers and chat servers. It receives response data from the API as input and generates a list of the acquired raw data as output.

[1583] Step 2:

[1584] The server converts the acquired historical data into a unified format. Specifically, it extracts the subject, body, sender, recipient, and date and time from email data, and the message content, sender, recipient, and timestamp information from chat messages. It receives a list of raw data as input and generates a list of standardized data as output.

[1585] Step 3:

[1586] The server saves the converted data in a database, which is a data storage means. Specifically, it inserts the data into an SQLite database and also checks for duplicates. It receives a list of standardized data as input and generates the number of records of the data stored in the database as output.

[1587] Step 4:

[1588] A user accesses a dedicated search interface and enters search criteria, such as search keywords and date ranges. The system takes the search criteria specified by the user as input and generates a query based on the search criteria as output.

[1589] Step 5:

[1590] The server generates an SQL query based on the search criteria it receives. Specifically, it creates an SQL statement that includes search keywords and date ranges. It takes the search criteria specified by the user as input and generates an executable SQL statement as output.

[1591] Step 6:

[1592] The server executes the generated SQL query against the database to search for the corresponding historical data. Specifically, it retrieves the relevant records from the database and sorts them in chronological order or by relevance. It receives the SQL statement as input and generates a list of search results as output.

[1593] Step 7:

[1594] The server uses an emotion recognition engine to analyze the user's emotions. Specifically, it analyzes the text and voice input by the user and determines whether the emotion is positive, negative, or neutral. It receives the user's text and voice data as input and generates the type of emotion as output.

[1595] Step 8:

[1596] The server adjusts search results based on the user's emotions. Specifically, it sorts the search results to prioritize the most important information. As input, it receives the emotion type and a list of search results, and as output, it generates a list of adjusted search results.

[1597] Step 9:

[1598] The server displays the adjusted search results in an appropriate presentation style, specifically, applying a vibrant design for positive emotions and a subdued design for negative emotions. The server receives the adjusted search results list as input and generates a search result screen to be displayed on the user's device as output.

[1599] Step 10:

[1600] The user can interact with the displayed search results and view more information. For example, they can click on a specific order history or support interaction history to view its contents. The system takes the displayed search results as input and displays a modal window or a new page with more information as output.

[1601] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1602] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1603] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1604] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1605] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1606] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1607] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1608] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1609] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1610] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1611] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1612] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1613] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1614] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1615] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1616] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1617] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1618] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1619] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1620] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1621] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1622] The following is further disclosed regarding the above embodiment.

[1623] (Claim 1)

[1624] A means of obtaining historical data from various communication tools;

[1625] means for converting the historical data into a unified format;

[1626] means for storing the data converted into the unified format in a database;

[1627] means for searching the database for corresponding history data based on search conditions specified by a user;

[1628] The system further includes means for visualizing the retrieved historical data to a user.

[1629] (Claim 2)

[1630] 10. The system of claim 1, further comprising means for sorting the search results in chronological order or by relevance.

[1631] (Claim 3)

[1632] 10. The system of claim 1, further comprising means for processing search requests from multiple users in parallel.

[1633] "Example 1"

[1634] (Claim 1)

[1635] A means for acquiring history information from various information transmission tools;

[1636] means for converting the history information into a unified format;

[1637] means for storing the information converted into the unified format in a data warehouse;

[1638] means for searching the data warehouse for corresponding history information based on search conditions designated by a user;

[1639] a means for visualizing the searched history information to a user;

[1640] a means for periodically obtaining historical information;

[1641] A means of checking for duplicates and storing information;

[1642] a means of accessing a dedicated search interface;

[1643] A means to enter search keywords, date ranges, and specific senders and recipients;

[1644] means for generating a condition-based query language;

[1645] A system that includes a means to further sort data in chronological order or by relevance after data acquisition.

[1646] (Claim 2)

[1647] 10. The system of claim 1, wherein the search results are sorted in chronological order or by relevance.

[1648] (Claim 3)

[1649] 10. The system of claim 1, wherein search requests from multiple users are processed in parallel.

[1650] "Application Example 1"

[1651] (Claim 1)

[1652] A means of obtaining historical data from various communication tools;

[1653] means for converting the historical data into a unified format;

[1654] means for storing the data converted into the unified format in a database;

[1655] means for searching the database for corresponding history data based on search conditions specified by a user;

[1656] means for visualizing the retrieved history data to a user;

[1657] a means for acquiring historical data through data collection software and interface software, and converting and storing the data in a unified format;

[1658] means for performing search and visualization through an application corresponding to the user's smart device;

[1659] A system that includes a means for automatically generating search criteria recommendations and prompts using a generative AI model.

[1660] (Claim 2)

[1661] 10. The system of claim 1, further comprising means for sorting the search results in chronological order or by relevance.

[1662] (Claim 3)

[1663] 10. The system of claim 1, further comprising means for processing search requests from multiple users in parallel.

[1664] "Example 2: Combining Emotion Engines"

[1665] (Claim 1)

[1666] means for acquiring communication history data from various information communication devices;

[1667] means for converting the communication history data into a standard format;

[1668] means for storing the data converted into the standard format in a storage device;

[1669] means for searching the storage device for corresponding communication history data based on search conditions designated by a user;

[1670] means for displaying the searched communication history data to a user;

[1671] means for recognizing a user's emotion;

[1672] means for tailoring search results based on the recognized sentiment;

[1673] means for changing the display format of search results based on the recognized emotion;

[1674] A system including:

[1675] (Claim 2)

[1676] 10. The system of claim 1, further comprising means for sorting the search results in chronological order or by relevance.

[1677] (Claim 3)

[1678] 10. The system of claim 1, further comprising means for processing search requests from multiple users simultaneously.

[1679] "Application example 2 when combining emotion engines"

[1680] (Claim 1)

[1681] A means for acquiring historical data from various communication means;

[1682] means for converting the historical data into a unified format;

[1683] a means for storing the data converted into the unified format in a data storage means;

[1684] means for searching the data storage means for corresponding history data based on search conditions designated by a user;

[1685] means for visualizing the retrieved history data to a user;

[1686] an emotion recognition engine that analyzes the user's emotions;

[1687] A means to tailor search results based on sentiment;

[1688] A system including a means for changing presentation style based on emotion.

[1689] (Claim 2)

[1690] 10. The system of claim 1, further comprising means for ordering the search results by time or relevance.

[1691] (Claim 3)

[1692] 10. The system of claim 1, further comprising means for processing query requests from a plurality of users in parallel. [Explanation of symbols]

[1693] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of obtaining historical data from various communication tools; means for converting the historical data into a unified format; means for storing the data converted into the unified format in a database; means for searching the database for corresponding history data based on search conditions specified by a user; The system further includes means for visualizing the retrieved historical data to a user.

2. The system of claim 1 further comprising means for sorting the search results in chronological order or by relevance.

3. 10. The system of claim 1, further comprising means for processing search requests from multiple users in parallel.

Images