system

Abstract

Provide a system. [Solution] A means of collecting data from multiple communication tools; a means for storing the collected data in a database; A means for analyzing the stored data using natural language processing algorithms; means for automatically classifying the analyzed data; means for attaching metadata to the classified data; A means for searching the database based on search words entered by the user on the search screen; a means of displaying search results in order of relevance; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's business environment, multiple communication tools are used, making it difficult to know which tool stores the information or files you need. As a result, users spend a lot of time trying to retrieve information, which reduces work efficiency. Another issue is that messages and files are scattered across different tools, making it difficult to centrally manage information. [Means for solving the problem]

[0005] To solve this problem, the present invention provides a system that collects data from multiple communication tools and stores it in a database. This system analyzes the stored data using a natural language processing algorithm and automatically classifies it. Furthermore, it adds metadata such as timestamps and sender information to the classified data, searches the database based on search terms entered by the user on a search screen, and displays the results in order of relevance. This allows users to quickly and efficiently obtain the information they need, improving work efficiency.

[0006] "Communication tools" are software or systems that allow users to exchange information, such as email, chat apps, and project management software.

[0007] "Data" refers to a collection of information including message text, attachments, timestamps, sender information, etc., sent and received via communication tools.

[0008] "Database" means an electronic repository for storing, managing, and retrieving collected data.

[0009] "Storage" refers to the process of recording acquired data in a database.

[0010] "Natural language processing algorithm" is a general term for machine learning methods and technologies used to analyze text data and perform keyword extraction and context understanding.

[0011] "Classification" is the process of organizing and separating data according to specific criteria or categories.

[0012] "Metadata" is supplementary information that accompanies data, including timestamps and sender information.

[0013] A "search word" is a keyword or phrase that a user enters to search for information.

[0014] "Sort by relevance" refers to sorting search results based on their relevance to the user's search word.

[0015] The term "system" refers to a combination of a series of hardware and software that executes the above means and functions in an integrated manner. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] MODE FOR CARRYING OUT THE INVENTION

[0038] This invention is a system that collects data from multiple communication tools, stores it in a database, analyzes the data using a natural language processing algorithm, and provides related information based on search terms. Specific embodiments of this system and the program processing procedures are described below.

[0039] 1. Data collection

[0040] First, the server gets access to Gmail, GChat, and other communication tools using their API keys or OAuth tokens, and then periodically checks for new messages and files.

[0041] For example, if a server collects data from Gmail, it uses the Gmail API to retrieve unread emails and extracts the email content, attachments, sent time (timestamp), and sender information. This extracted data is then stored in a database in a data format such as JSON.

[0042] 2. Data Analysis

[0043] The server then analyzes the stored data using natural language processing algorithms (NLP), such as SpaCy or BERT, to extract key keywords and context from the content of the message body and attachments. This analysis identifies criteria and categories for classifying the data.

[0044] The server automatically categorizes each message or file based on extracted keywords and context, and complements the data with metadata such as timestamps and sender information.

[0045] 3. Search and display in the user interface

[0046] Users use a dedicated search screen to enter search terms to find the information they need. The server queries the database based on the search terms and retrieves related messages and files. The retrieved search results are sorted by relevance and displayed in an easy-to-read format for the user.

[0047] For example, if a user searches for "Project X report," the server will search for report messages related to Project X and display them in order of relevance, along with metadata such as timestamps and sender information. The search results include the message body, links to attachments, time of sending, and sender information.

[0048] Specific examples

[0049] As a concrete example, let's consider the case where a server collects data from G Chat. In this case, the server first obtains new messages using the G Chat API. The server then extracts the contents, attachments, timestamps, sender information, etc. from the obtained messages and stores them in a database.

[0050] The server then analyzes this data and categorizes it into categories, such as "progress reports" or "meeting minutes." If a user searches for "meeting minutes," messages and files related to that search term are displayed. The server displays results based on chronology and importance, allowing users to quickly access the information they need.

[0051] This system allows users to centrally manage what information has been exchanged with which tools, and efficiently obtain the information they need, greatly improving work efficiency.

[0052] The processing flow will be explained below.

[0053] Step 1: Collect data

[0054] 1. The server obtains access rights using the API key or OAuth token of each communication tool (e.g., Gmail, GChat).

[0055] 2. The server calls the API of each tool at regular intervals to check whether there are any new messages or files.

[0056] 3. The server extracts the following information from the retrieved message or file:

[0057] Message body

[0058] Attachments

[0059] timestamp (time of sending)

[0060] Sender information (email address, name, etc.)

[0061] 4. The server stores the extracted data in a database in JSON or other data formats.

[0062] Step 2: Analyze the data

[0063] 1. The server analyzes messages and attachments stored in a database using natural language processing algorithms.

[0064] Example: Use SpaCy or BERT to extract key keywords and context.

[0065] 2. The server automatically classifies each message or file based on the extracted keywords.

[0066] Example: Classify data related to Project X based on the keyword "Project X".

[0067] 3. The server adds metadata such as timestamps and sender information to the classified data.

[0068] Step 3: Perform a search

[0069] 1. The user accesses a dedicated search screen and enters a search term.

[0070] For example, enter "Project X Report."

[0071] 2. The server queries the database based on the entered search term.

[0072] Example: Search for "Project X Report" to get relevant data.

[0073] Step 4: Formatting and displaying search results

[0074] 1. The server sorts the retrieved search results by relevance.

[0075] Example: sorting search results based on relevance score.

[0076] 2. The server formats the search results in a format that is easy for the user to understand.

[0077] Each result includes the message body, attachments (with links), timestamps, and sender information.

[0078] 3. The user reviews the organized search results and accesses the information they need.

[0079] Through these steps, users can centrally manage information that is scattered across multiple communication tools and efficiently obtain the information they need.

[0080] Example 1

[0081] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0082] In today's world, it is common for people to use multiple information transmission tools for business and communication. However, it is difficult to centrally manage and analyze data from these information transmission tools and efficiently search and acquire necessary information. In particular, there is a demand for a system that can uniformly process data in different formats and quickly extract highly relevant information. The objective of this invention is to provide a system that solves these data management and analysis problems and enables users to efficiently acquire necessary information.

[0083] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0084] In this invention, the server includes means for collecting data from multiple information transmission tools, means for storing the collected data in a data management device, means for analyzing the stored data using a natural language processing algorithm, means for automatically classifying the analyzed data, means for adding metadata to the classified data, means for searching the data management device based on search terms entered by an information user on a search screen, means for displaying search results in order of relevance, means for using SpaCy or BERT as a natural language processing algorithm, means for storing the collected data in JSON format, and means for periodically checking for new messages and files from the information transmission tools. This enables information users to efficiently search and retrieve data centrally managed by different information transmission tools.

[0085] An "information communication tool" is a system for sending and receiving messages or files in digital form, such as email, chat applications, or online messaging systems.

[0086] A "data management device" is a database or storage system for storing, managing, and searching collected data.

[0087] "Natural language processing algorithms" are algorithms used to analyze text data, extract keywords, understand context, classify categories, etc. Examples include SpaCy and BERT.

[0088] "Metadata" refers to attribute information of data, and includes, for example, the date and time of data creation, sender information, related categories, and the like.

[0089] The "search screen" is a user interface that allows an information user to input a search word and execute a search query on the data management device.

[0090] "Relevance order" refers to the order in which search results are sorted in order of the most relevant to the information user's search word.

[0091] "SpaCy" is an open source Python library for implementing advanced natural language processing.

[0092] "BERT" is a transformer-based natural language processing model and an algorithm for performing highly accurate text analysis.

[0093] "JSON format" is an abbreviation for JavaScript (registered trademark) Object Notation, and is a data format that expresses structured data in a text format that is easy for humans to read and machine processing.

[0094] "Periodic check for new messages and files" refers to the process where the server calls the API at regular intervals to check for newly added messages and files.

[0095] MODE FOR CARRYING OUT THE INVENTION

[0096] The present invention provides a system that collects data from multiple information transmission tools, stores it in a data management device, analyzes the data using a natural language processing algorithm, and provides related information based on search terms. This system allows information users to efficiently and quickly obtain the information they need.

[0097] First, the server collects data from multiple communication tools, including email, chat applications, and online messaging systems. In this process, the server obtains API access using each tool's API key or OAuth token. For example, it periodically checks for new messages and files using the APIs of Gmail and chat applications to obtain data. The obtained data is then stored in a data management device in JSON format.

[0098] The server then analyzes the stored data using natural language processing algorithms. This analysis uses advanced natural language processing libraries such as SpaCy and BERT. The server tokenizes the text data and automatically classifies it by extracting keywords and performing contextual analysis. Key keywords and context are extracted from the content of messages and files, and they are then classified into categories such as "progress report" or "meeting minutes." Metadata such as timestamps and sender information is also added to the data.

[0099] The information user then uses a dedicated search screen to enter search terms to find the information they need. The device sends the search terms to the server, which then queries the data management device to retrieve related messages and files. The retrieved search results are sorted by relevance and displayed in a format that is easy for the information user to view. For example, if the information user searches for "Project X report," the server will display a list of reports and emails related to Project X, providing the message body, links to attachments, the time of sending, sender information, and more.

[0100] As a concrete example, let us consider the case where a server collects data from a chat application. The server first uses the chat application's API to obtain new messages. It extracts the content, attachments, timestamps, and sender information from the obtained messages and stores them in a data management device. The server then analyzes this data and classifies it into categories such as "progress reports" and "meeting minutes." When an information user searches for "meeting minutes," messages and files related to that search term are displayed. The server displays the search results in chronological order and by importance, allowing the information user to quickly access the information they need.

[0101] Examples of prompts for generative AI models include:

[0102] How can I get unread emails using the Email API?

[0103] "Explain how to extract keywords from text using a natural language processing library."

[0104] "What's the best way to construct a search query?"

[0105] In this way, by using this system, information users can efficiently search and retrieve data that is centrally managed across multiple information transmission tools, significantly improving work efficiency.

[0106] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0107] Step 1: Collect data

[0108] The server establishes API access. Specifically, it obtains API access rights using the API key or OAuth token of an information transmission tool such as Gmail or a chat application.

[0109] Input: API key or OAuth token of the information transmission tool

[0110] How it works: Obtain an access token through the API authentication process

[0111] Output: Access token

[0112] The server acquires data by collecting new messages and files from each information transmission tool via API.

[0113] Input: Access token, API endpoint

[0114] What it does: For example, call an endpoint that uses the Gmail API to get unread emails.

[0115] Output: Collected data (message body, attachments, timestamp, sender information)

[0116] The server saves the collected data in a data management device. The collected data is stored in the data management device in JSON format.

[0117] Input: Collected data (JSON format)

[0118] Action: Executes an INSERT statement to the database and saves the data.

[0119] Output: Data stored in the data management device

[0120] Step 2: Analyze the data

[0121] The server applies natural language processing algorithms, using libraries such as SpaCy and BERT, to analyze the stored data.

[0122] Input: Text data stored in the data management device

[0123] Operation: Tokenizes text data, extracts keywords, and performs contextual analysis

[0124] Output: Analysis results (extracted keywords and context information)

[0125] The server then categorizes the data based on the analysis results, categorizing messages and files based on extracted keywords and context.

[0126] Input: Analysis results

[0127] Behavior: For example, classifying data into categories such as "progress reports" and "meeting minutes"

[0128] Output: Classified data

[0129] The server adds metadata, such as a timestamp and sender information, to the data.

[0130] Input: Classified data, timestamp information, caller information

[0131] Behavior: Adds metadata to each data entry

[0132] Output: Data with metadata

[0133] Step 3: Search and display in the user interface

[0134] Users use a dedicated search screen to input search terms to find the information they need.

[0135] Input: Search word

[0136] Operation: The device sends the search word to the server.

[0137] Output: Search request to the server

[0138] The server performs a database query, querying the data management device based on the entered search words.

[0139] Input: Search word

[0140] Operation: Searches for the relevant data using the SQL SELECT statement

[0141] Output: Query results (associated messages and files)

[0142] The server formats the search results in order of relevance, making the retrieved data easy for information users to view.

[0143] Input: Query results

[0144] What it does: Sorts search results by relevance and formats them in HTML.

[0145] Output: Formatted search results

[0146] The terminal displays the search results, ie, displays the search results returned from the server to the user.

[0147] Input: Formatted search results

[0148] Action: Show search results

[0149] Output: Search results displayed to the user

[0150] In this way, by processing each step sequentially, this system enables information users to efficiently search and retrieve data that is centrally managed across multiple information transmission tools.

[0151] (Application example 1)

[0152] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0153] Previously, maintenance information for machinery and equipment within a factory was managed separately using multiple communication methods, making centralized management of the information difficult. Furthermore, searching and analyzing maintenance information required a great deal of time and effort, making efficient maintenance management difficult. This resulted in delays in responding to machine breakdowns and malfunctions, leading to reduced production efficiency.

[0154] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0155] In this invention, the server includes means for collecting data from multiple communication means, means for storing the collected data in an information storage device, and means for analyzing the stored data using a natural language processing algorithm, thereby enabling the centralized collection and analysis of information related to the maintenance of factory equipment and the prompt provision of necessary information.

[0156] "Multiple means of communication" refers to a variety of digital communication tools such as email, chat, and maintenance logs.

[0157] "Data collection method" refers to the function of periodically obtaining new messages and files using APIs and OAuth tokens of multiple communication methods.

[0158] "Information storage device" refers to a digital storage medium such as a database or cloud storage for storing collected data.

[0159] "Natural language processing algorithm" refers to technology that analyzes text data using methods such as SpaCy and BERT and extracts keywords and contextual information.

[0160] "Automatic classification means" refers to a function that automatically classifies analyzed data based on specific criteria or categories.

[0161] "Attribute information" refers to metadata such as timestamps and sender information that is added to data.

[0162] "Search instructions" refer to keywords or phrases that a user enters into a search screen.

[0163] "Search means" refers to a function that queries an information storage device based on a search instruction entered by a user to retrieve relevant data.

[0164] "Means for displaying in order of relevance" refers to a function for sorting search results in order of relevance and displaying them to the user.

[0165] "Information on the maintenance of factory equipment" refers to data related to the maintenance, inspection, and repair of machinery and equipment within a factory.

[0166] The present invention relates to a system for efficiently managing information relating to the maintenance of factory equipment and for quickly providing necessary information. Specific embodiments of the present invention will be described below.

[0167] First, the server collects data from multiple communication methods required within the factory (e.g., email, chat, maintenance logs). The data is acquired using the API of each communication method and stored in an information storage device (database or cloud storage).

[0168] The server then analyzes the stored data using natural language processing algorithms. Specifically, it uses natural language processing libraries such as SpaCy and BERT to analyze the text data and extract keywords and contextual information. This allows for automatic classification of the data and the addition of attribute information (such as timestamps and sender information).

[0169] Users input search instructions into a search screen using a display or tablet installed on a factory robot. Based on this, the server queries the information storage device and retrieves relevant data. Search results are displayed in order of relevance and presented in a format that is intuitively easy for users to understand.

[0170] For example, if a factory robot searches for "Machine A's maintenance history for the past six months," the system quickly displays detailed historical information, including timestamps and maintenance details. Similarly, if a user searches for "error messages," the system also displays related repair procedures and contact information.

[0171] This system centrally manages information related to factory equipment maintenance and provides necessary information quickly, which speeds up responses to machine breakdowns and malfunctions and improves production efficiency.

[0172] Examples of prompts for a generative AI model might include:

[0173] "We want to develop a system that collects maintenance log data, analyzes its contents, and provides maintenance history and error information. What architecture and technology would be suitable?"

[0174] In this manner, specific elements and procedures are described in detail to clearly provide specific embodiments of the invention.

[0175] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0176] Step 1:

[0177] The server collects data from multiple communication methods within the factory (email, chat, maintenance logs). Specifically, it uses APIs and OAuth tokens to obtain access to each communication method and periodically checks whether there are any new messages or files. The input requires the API key or OAuth token for each communication method, and the output is the collected message and file data.

[0178] Step 2:

[0179] The server stores the collected data in an information storage device (database). Specifically, it extracts attribute information such as the contents of the collected messages and files, timestamps, and sender information, converts it into an appropriate format (e.g., JSON format), and stores it in the database. The input is the data collected in step 1, and the output is the data stored in the information storage device.

[0180] Step 3:

[0181] The server analyzes the data stored in the information storage device using natural language processing algorithms (e.g., SpaCy or BERT). Specifically, it analyzes the text data to extract key keywords and contextual information, and then classifies the data based on that. The input is the unanalyzed data stored in the information storage device, and the output is the analyzed data and its keyword information.

[0182] Step 4:

[0183] The server adds attribute information, such as a timestamp and sender information, to the analyzed data. This allows the analyzed data to be classified in more detail and supplemented with related information. The input is the analyzed data, and the output is the analyzed data with the attribute information added.

[0184] Step 5:

[0185] The user inputs search instructions using a tablet or display installed on the factory robot. Specifically, they input keywords or phrases (e.g., "Maintenance history of machine A") into the search screen. The input is the search instruction entered by the user, and the output is the search instruction itself.

[0186] Step 6:

[0187] The server queries the information storage device based on the search instructions entered by the user. Specifically, it finds data corresponding to the search keywords and retrieves related data. The input is the search instructions entered by the user, and the output is the related data as the search results.

[0188] Step 7:

[0189] The server sorts the retrieved search results in order of relevance and provides them to the user. Specifically, it formats the search results in order of relevance and displays them on the display in a format that is intuitively easy for the user to understand. The input is related data as search results, and the output is a display of the formatted search results.

[0190] Step 8:

[0191] The user selects the required information from the displayed search results and checks the details. Specifically, the user clicks on the relevant information from the list of search results to open the details screen. The input is the list of search results, and the output is the selected detailed information.

[0192] The above steps will realize a system that efficiently collects and manages information related to factory equipment maintenance and quickly provides the necessary information. This system will enable a quick response to machine breakdowns and malfunctions, contributing to improved production efficiency.

[0193] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0194] MODE FOR CARRYING OUT THE INVENTION

[0195] This invention is a system that combines a system that collects data from multiple communication tools, stores the data in a database, and analyzes and classifies it using natural language processing algorithms with an emotion engine that recognizes the user's emotions. Specific embodiments of this system and the program processing procedures are described below.

[0196] 1. Data collection

[0197] The server obtains access using API keys or OAuth tokens from Gmail, GChat, or other communication tools. The server uses these APIs to periodically check for new messages or files. For example, to collect data from Gmail, the server uses the Gmail API to retrieve unread emails and extract the email content, attachments, sent time (timestamp), and sender information. This extracted data is then stored in a database in a data format such as JSON.

[0198] 2. Data Analysis

[0199] The server then analyzes the stored data using natural language processing algorithms (NLP), such as SpaCy or BERT, to extract key keywords and context from the content of message bodies and attachments. This analysis identifies criteria and categories for classifying the data. The server then automatically classifies each message or file based on the extracted keywords and context. Furthermore, the server complements the data with metadata such as timestamps and sender information.

[0200] 3. User Emotion Recognition

[0201] The server uses an emotion engine to recognize the user's emotions when the user accesses the search screen and enters a search term. The emotion engine evaluates the user's emotional state through facial recognition, voice analysis, and text analysis. For example, when a user searches for "Project X report," the emotion engine analyzes the user's voice tone, facial expression, and the tone of the text entered to understand the user's current emotional state.

[0202] 4. Tailor search results based on sentiment

[0203] The server adjusts the order of search results based on the user's emotions, as determined by the emotion engine. For example, if the user is feeling stressed, the server prioritizes relevant results and reduces unnecessary information. If the user is feeling relaxed, the server also displays detailed explanations and related materials.

[0204] 5. Search Results Display and Optimization

[0205] Users can check the search results provided by the server. The server continuously analyzes the user's reactions using an emotion engine and optimizes the display of search results from the next time. For example, if a user has a positive reaction to a particular search result, the server will display results based on that reaction when a similar search is performed.

[0206] Specific examples

[0207] For example, if a user searches for "Project X meeting minutes," the server will search for meeting minutes messages related to Project X and display them in order of relevance along with metadata such as timestamps and sender information. At this time, the emotion engine will adjust the display order of search results as needed based on the user's emotional state. The emotion engine also determines whether the user is satisfied with the search results, which will be used to improve the accuracy of search results in future searches.

[0208] This system allows users to centrally manage what information has been exchanged with which tools, enabling them to efficiently obtain the information they need. In addition, by combining it with an emotion engine, it is possible to display information optimally according to the user's emotional state, further improving work efficiency.

[0209] The processing flow will be explained below.

[0210] Step 1: Collect data

[0211] 1. The server obtains access rights using the API key or OAuth token of each communication tool (e.g., Gmail, GChat).

[0212] 2. The server calls the API of each tool at regular intervals to check whether there are any new messages or files.

[0213] 3. The server extracts the following information from the retrieved message or file:

[0214] Message body

[0215] Attachments

[0216] timestamp (time of sending)

[0217] Sender information (email address, name, etc.)

[0218] 4. The server stores the extracted data in a database in JSON or other data formats.

[0219] Step 2: Analyze the data

[0220] 1. The server analyzes messages and attachments stored in a database using natural language processing algorithms.

[0221] For example, use SpaCy or BERT to extract key keywords and context.

[0222] 2. The server automatically classifies each message or file based on the extracted keywords.

[0223] For example, data related to Project X may be classified based on the keyword "Project X."

[0224] 3. The server adds metadata such as timestamps and sender information to the classified data.

[0225] Step 3: Acquire data for emotion recognition

[0226] 1. The user accesses a dedicated search screen and prepares to enter a search term.

[0227] 2. The device collects the user's facial expression and voice data when the user enters a search term.

[0228] For example, a webcam and microphone are used to capture the user's facial expressions and voice tone.

[0229] 3. The server analyzes the collected emotion data using an emotion engine.

[0230] For example, facial recognition software and voice analysis tools are used to determine a user's emotional state.

[0231] Step 4: Perform sentiment-based search

[0232] 1. The user enters a search term into a dedicated search screen.

[0233] For example, enter "Project X Report."

[0234] 2. The server queries the database using the emotion data from the emotion engine along with the entered search term.

[0235] For example, if you search for "Project X report" and the user is feeling stressed, the system will prioritize the most relevant results.

[0236] Step 5: Formatting and displaying search results

[0237] 1. The server sorts the retrieved search results by relevance.

[0238] For example, the analysis results from the emotion engine are taken into account to display more relevant results at the top.

[0239] 2. The server formats the search results in a format that is easy for the user to understand.

[0240] Each result includes the message body, attachments (with links), timestamps, and sender information.

[0241] 3. The user reviews the organized search results and accesses the information they need.

[0242] Step 6: Collect and optimize responses to search results

[0243] 1. The device collects the user's reactions as they check the search results.

[0244] For example, it monitors the user's facial expressions, voice tone, clicking behavior, etc.

[0245] 2. The server analyzes the reaction data with an emotion engine to evaluate how users reacted to a particular search result.

[0246] 3. The server will optimize the search results displayed from the next time onwards based on the analysis results.

[0247] For example, the search algorithm can be adjusted to prioritize results that satisfy users.

[0248] This allows the user to efficiently obtain the necessary information and enables optimal information display according to the user's emotional state.

[0249] Example 2

[0250] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0251] In today's business environment, a large amount of information is exchanged through multiple communication tools, creating a need for centralized data management and efficient information search. However, due to a lack of optimization of search results that take into account the user's emotional state, it is difficult to provide appropriate information according to the user's state. This leads to problems such as reduced work efficiency and increased stress for users.

[0252] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0253] In this invention, the server includes means for collecting data from multiple communication tools, means for saving the collected data in a database, means for analyzing the saved data using a natural language processing algorithm, means for automatically classifying the analyzed data, means for adding metadata to the classified data, means for searching the database based on search words entered by a user on a search screen, means for displaying search results in order of relevance, means for using an emotion engine that recognizes the user's emotions when the user accesses the search screen, means for adjusting the display order of search results based on the emotion information recognized by the emotion engine, and means for continuously analyzing the user's reactions using the emotion engine and optimizing the display of search results from the next time onwards. This enables centralized management of information and provision of appropriate information according to the user's emotional state.

[0254] "Multiple communication tools" refers to various means by which users exchange messages and data electronically, such as email and chat applications.

[0255] "Means for collecting data" refers to software and hardware components used to obtain message and file information from designated communication tools.

[0256] "Means for storing data in a database" refers to a storage device or management system for permanently storing collected data.

[0257] "Natural language processing algorithms" refer to machine learning and language models that analyze collected text data and extract context and meaning.

[0258] "Means for automatically classifying data" refers to a function that divides data into specific categories or tags based on the results of analysis.

[0259] "Metadata" refers to additional information about a message or file (e.g., timestamps and originator information).

[0260] A "search screen" refers to an interface that allows a user to input search keywords and search for information in a database.

[0261] "Emotion engine" refers to technology that performs facial recognition, voice analysis, and text analysis of a user to assess their emotional state.

[0262] "Means for adjusting the display order of search results" refers to a function for sorting search results in an appropriate order based on the user's emotional state.

[0263] "Means for continuously analyzing user responses" refers to technology that collects user search activity and responses to the results, and optimizes the display of search results from the next time onwards.

[0264] MODE FOR CARRYING OUT THE INVENTION

[0265] This invention is a system that combines a system that collects data from multiple communication tools, stores the data in a database, and analyzes and classifies the data using a natural language processing algorithm with an emotion engine that recognizes the user's emotions. Specific embodiments for implementing this system will be described below.

[0266] 1. Data collection

[0267] The server obtains access rights from multiple communication tools, such as Gmail and chat tools, using API keys and OAuth tokens. It periodically checks for new messages and files using the Gmail API and chat API. For example, when collecting data from Gmail, the server retrieves unread emails and extracts the email content, attachments, sent date and time, and sender information. This extracted data is stored in a database in a data format such as JSON.

[0268] 2. Data Analysis

[0269] The server then analyzes the stored data using natural language processing algorithms, particularly tools like SpaCy and BERT. These tools extract key keywords and context from the content of messages and attachments. This analysis determines the criteria and categories for categorizing the data. The server automatically classifies each message or file based on the extracted keywords and context. It also adds metadata such as timestamps and sender information to complement the data.

[0270] 3. User Emotion Recognition

[0271] When a user accesses the search screen from their device and enters a search term, the server uses an emotion engine to recognize the user's emotion. The emotion engine evaluates the user's emotional state through facial recognition, voice analysis, and text analysis. For example, when a user searches for "Project X Report," the server analyzes the user's voice tone, facial expression, and the tone of the text entered to understand the user's current emotional state.

[0272] 4. Tailor search results based on sentiment

[0273] The server adjusts the display order of search results based on the emotional information recognized by the emotion engine. For example, if the user is feeling stressed, the server will prioritize relevant results and reduce unnecessary information. If the user is feeling relaxed, the server will also display detailed explanations and related materials.

[0274] 5. Search Results Display and Optimization

[0275] The server displays the adjusted search results on the user's device. The user can then review the displayed search results. Furthermore, the server continuously analyzes the user's reactions using an emotion engine to optimize the display of search results from the next time onwards. For example, if a user has a positive reaction to a particular search result, the server will display results based on that reaction when a similar search is performed.

[0276] Examples of concrete examples and prompts

[0277] For example, consider the case where a user searches for "Project X meeting minutes." The server searches the database for messages of meeting minutes related to Project X and sorts them in order of relevance along with metadata such as timestamps and sender information. The emotion engine then adjusts the display order of the search results as needed based on the user's emotional state. The emotion engine also analyzes the user's reaction to the search results and optimizes the display of search results from the next time onwards.

[0278] An example of a prompt sentence might be "Please search for Project X meeting minutes."

[0279] This system allows users to centrally manage what information has been exchanged using which tools, enabling them to efficiently obtain the information they need. In addition, the introduction of an emotion engine makes it possible to provide optimal information according to the user's emotional state, significantly improving work efficiency.

[0280] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0281] Step 1: Collect data

[0282] 1.1 The server obtains API keys and OAuth tokens from multiple communication tools such as Gmail and chat tools. The input is the authentication information of each tool, and the output is the obtained access rights information.

[0283] 1.2 The server uses the obtained access right information to access each tool and periodically checks whether new messages or files exist. The input is the access right information, and the output is the new message or file information.

[0284] 1.3 The server retrieves unread emails using the Gmail API and extracts the email content, attachments, sent date and time, and sender information. The input is access rights information and new message information, and the output is the extracted email data.

[0285] 1.4 The server saves the extracted data in a database in JSON format. The input is the extracted email data, and the output is the data saved in the database.

[0286] Step 2: Analyze the data

[0287] 2.1 The server analyzes the stored data based on a natural language processing algorithm, where the input is the stored data and the output is the analysis result.

[0288] 2.2 The server uses natural language processing tools such as SpaCy or BERT to extract key keywords and context from the content of the message body and attachments. The input is the stored data, and the output is the extracted keywords and context.

[0289] 2.3 The server automatically classifies each message or file based on the extracted keywords and context. The input is the extracted keywords and context, and the output is the classified data.

[0290] 2.4 The server complements the data by adding metadata such as timestamps and sender information. The input is the classified data, and the output is the data with the metadata added.

[0291] Step 3: Recognizing user emotions

[0292] 3.1 The user accesses the search screen from their device and enters a search term. The input is the search term, and the output is the transmission of the search term.

[0293] 3.2 The server passes the user's input to the emotion engine. The input is the search word, and the output is the input to the emotion engine.

[0294] 3.3 The emotion engine evaluates the user's emotional state through facial recognition, voice analysis, and text analysis. The input is the search term and the user's biometric information, and the output is the user's emotional state.

[0295] 3.4 For example, when a user searches for "Project X report," the emotion engine analyzes the user's tone of voice, facial expression, and the tone of the text entered to understand the user's current emotional state. The input is the search word and the user's biometric information, and the output is the analyzed emotional state.

[0296] Step 4: Tailor search results based on sentiment

[0297] 4.1 The server adjusts the display order of search results based on the user's emotional information recognized by the emotion engine. The input is the user's emotional information, and the output is the adjusted search results.

[0298] 4.2 For example, if a user is feeling stressed, the server prioritizes relevant results and reduces unnecessary information. The input is the user's emotional information, and the output is the prioritized search results.

[0299] 4.3 If the user is relaxed, detailed explanations and related materials are also displayed. The input is the user's emotional information, and the output is search results with added detailed information.

[0300] Step 5: Displaying and optimizing search results

[0301] 5.1 The server displays the tailored search results on the user's terminal. The input is the tailored search results and the output is the results displayed on the terminal.

[0302] 5.2 The user can confirm the search results displayed on the terminal. The input is the displayed search results, and the output is the user's confirmation.

[0303] 5.3 The server analyzes user responses to search results (clicks, time spent, etc.) using an emotion engine. The input is the user response, and the output is the analysis result.

[0304] 5.4 The server will optimize the display of search results from the next time onwards based on the analysis results. The input is the analysis results, and the output is the optimized search result display format.

[0305] This series of processing steps allows the system to provide optimal information according to the user's emotional state, allowing the user to efficiently obtain the information they need and improve their work efficiency.

[0306] (Application example 2)

[0307] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0308] In recent years, brick-and-mortar stores have been required to provide services that respond promptly to customer needs, but it is difficult to grasp the emotional state of customers in real time, making it difficult to provide optimal customer service and product recommendations. Furthermore, there is a demand for methods to improve customer satisfaction by utilizing past communication data, but there is a lack of efficient ways to do this. There is an urgent need to provide a system that improves customer experience by resolving these issues.

[0309] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0310] In this invention, the server includes means for collecting data from multiple communication tools, means for saving the collected data in a database, means for analyzing the saved data using a natural language processing algorithm, means for automatically classifying the analyzed data, means for adding metadata to the classified data, means for searching the database based on search words entered by a user on a search screen, means for displaying search results in order of relevance, means for recognizing the user's emotions in real time, and means for adjusting the display order of information based on the recognized emotions. This makes it possible to grasp the emotional state of customers in real time and provide optimal customer service and product suggestions based on past communication data.

[0311] 1. "Communication tools" is a general term for software and hardware used to send and receive information via digital media, such as email and chat applications.

[0312] 2. "Database" means a data management system that efficiently manages collected data and allows for searching and analysis as needed.

[0313] 3. "Natural language processing algorithm" is a general term for algorithms that analyze text data and understand and process its meaning.

[0314] 4. "Classification" is the process of grouping analyzed data based on specific criteria.

[0315] 5. "Metadata" is a general term for attribute information and additional information related to data, which is added to facilitate data search and management.

[0316] 6. "Search screen" is an interface that allows users to enter keywords and search for information in the database.

[0317] 7. "Relevance" is a criterion for sorting search results based on the degree of match or importance with the keywords entered by the user.

[0318] 8. "Emotion recognition means" is a general term for technologies and devices that analyze a user's facial expressions and vocal tone to understand their emotional state.

[0319] 9. "Adjusting the information display order" is the process of changing the order of displayed information based on perceived user sentiment.

[0320] This invention is a system for grasping the emotional state of customers in real time in a physical store and providing optimal customer service and product recommendations. This system collects data from multiple communication tools and analyzes it with a natural language processing algorithm to grasp the user's emotional state in real time and adjust the order in which information is displayed based on this data.

[0321] First, the server collects data from communication tools such as Gmail and chat apps. The server uses the API keys or OAuth tokens of these tools to gain access and periodically checks for new messages and files. The collected data is stored in a database in a data format such as JSON, allowing salespeople to quickly reference past interactions with customers.

[0322] The server then analyzes the stored data using natural language processing algorithms. Generative AI models like SpaCy and BERT are used to extract key keywords and context from the content of messages and attachments. This analysis identifies criteria or categories for classifying the data. Classification tags and metadata are then added to the analyzed data.

[0323] Additionally, the system employs emotion recognition to recognize users' emotions in real time. This is achieved by using the camera and microphone built into the smart glasses to analyze the customer's facial expressions and tone of voice. For emotion recognition, the system uses the Hugging Face Transformers library to evaluate their emotional state. This emotion data is stored in the cloud and can be used to trigger specific actions.

[0324] The system also provides a means to adjust the order in which information is displayed based on emotions. For example, if a customer is feeling stressed, relevant search results will be displayed first, reducing unnecessary information. On the other hand, if the customer is relaxed, detailed explanations and related materials will also be displayed. This adjustment makes it possible to provide the optimal service according to the customer's state.

[0325] The following specific examples are given as examples.

[0326] Example 1: When a customer searches for "Project X meeting minutes"

[0327] The server searches for meeting minutes related to Project X and displays them in order of relevance along with timestamps and sender information. The emotion engine adjusts the display order according to the user's emotional state and provides detailed information as needed.

[0328] Example 2: When a customer is stressed

[0329] Prompt: "What is the appropriate way to respond when a customer is stressed?"

[0330] Example output: "Respond simply and quickly, and refrain from suggesting new products."

[0331] Example 3: When the customer is relaxed

[0332] Prompt: "What is the appropriate way to respond when a customer is relaxed?"

[0333] Sample output: "Provides detailed product descriptions and offers additional options."

[0334] This will enable customer service in physical stores to respond more quickly to the emotional state of customers and provide greater satisfaction.

[0335] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0336] Step 1:

[0337] The server obtains access rights using API keys or OAuth tokens from communication tools such as Gmail and chat apps. The input is the authentication information for each communication tool, and the output is the access rights to these tools. Specifically, the server periodically checks for and collects unread emails using the Gmail API.

[0338] Step 2:

[0339] The server collects new messages and files from the communication tools it has gained access to and stores them in a database in an appropriate format, such as JSON. The input is the data of the new messages and files, and the output is the data stored in the database. Specifically, timestamps and sender information are also stored in the database at the same time.

[0340] Step 3:

[0341] The server analyzes the stored data using natural language processing algorithms (e.g., SpaCy or BERT). The input is the text data stored in the database, and the output is the analysis results, including key keywords and contextual information. Specifically, meaning is extracted from the contents of the message body and attachments and automatically classified.

[0342] Step 4:

[0343] The server adds classification tags and metadata to the analyzed data. The input is the main keywords and context information from the analysis results, and the output is data with the classification tags and metadata added. Specifically, category information, timestamp, and sender information are added to each data.

[0344] Step 5:

[0345] A user enters keywords into a search screen, and the server searches the database based on those keywords. The input is the search keyword entered by the user, and the output is a list of related data. Specifically, the server extracts highly relevant data from the searched data.

[0346] Step 6:

[0347] The server sorts the search results in order of relevance and displays them to the user. The input is a list of searched data, and the output is the search results sorted in order of relevance. Specifically, the list is displayed taking into account related metadata.

[0348] Step 7:

[0349] The system grasps the user's emotional state in real time through a camera and microphone connected to a device (such as smart glasses). The input is camera video and microphone audio, and the output is the analysis result of the user's emotional state. Specifically, it uses the Hugging Face Transformers library to recognize emotions from facial expressions and voice tone.

[0350] Step 8:

[0351] The server adjusts the display order of information based on the recognized emotional state. The input is the analysis result of the user's emotional state and the search results, and the output is the adjusted display order of information. Specifically, if the user is feeling stressed, highly relevant information is displayed first, and if the user is relaxed, additional detailed information is displayed.

[0352] Step 9:

[0353] The server continuously analyzes customer responses using an emotion engine to optimize the display of search results from the next time onwards. The input is customer response data, and the output is search results in an optimized display order. Specifically, the next display is improved based on search results that have generated positive responses.

[0354] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0355] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0356] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0357] [Second embodiment]

[0358] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0359] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0360] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0361] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0362] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0363] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0364] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0365] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0366] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0367] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0368] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0369] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0370] MODE FOR CARRYING OUT THE INVENTION

[0371] This invention is a system that collects data from multiple communication tools, stores it in a database, analyzes the data using a natural language processing algorithm, and provides related information based on search terms. Specific embodiments of this system and the program processing procedures are described below.

[0372] 1. Data collection

[0373] First, the server gets access to Gmail, GChat, and other communication tools using their API keys or OAuth tokens, and then periodically checks for new messages and files.

[0374] For example, if a server collects data from Gmail, it uses the Gmail API to retrieve unread emails and extracts the email content, attachments, sent time (timestamp), and sender information. This extracted data is then stored in a database in a data format such as JSON.

[0375] 2. Data Analysis

[0376] The server then analyzes the stored data using natural language processing algorithms (NLP), such as SpaCy or BERT, to extract key keywords and context from the content of the message body and attachments. This analysis identifies criteria and categories for classifying the data.

[0377] The server automatically categorizes each message or file based on extracted keywords and context, and complements the data with metadata such as timestamps and sender information.

[0378] 3. Search and display in the user interface

[0379] Users use a dedicated search screen to enter search terms to find the information they need. The server queries the database based on the search terms and retrieves related messages and files. The retrieved search results are sorted by relevance and displayed in an easy-to-read format for the user.

[0380] For example, if a user searches for "Project X report," the server will search for report messages related to Project X and display them in order of relevance, along with metadata such as timestamps and sender information. The search results include the message body, links to attachments, time of sending, and sender information.

[0381] Specific examples

[0382] As a concrete example, let's consider the case where a server collects data from G Chat. In this case, the server first obtains new messages using the G Chat API. The server then extracts the contents, attachments, timestamps, sender information, etc. from the obtained messages and stores them in a database.

[0383] The server then analyzes this data and categorizes it into categories, such as "progress reports" or "meeting minutes." If a user searches for "meeting minutes," messages and files related to that search term are displayed. The server displays results based on chronology and importance, allowing users to quickly access the information they need.

[0384] This system allows users to centrally manage what information has been exchanged with which tools, and efficiently obtain the information they need, greatly improving work efficiency.

[0385] The processing flow will be explained below.

[0386] Step 1: Collect data

[0387] 1. The server obtains access rights using the API key or OAuth token of each communication tool (e.g., Gmail, GChat).

[0388] 2. The server calls the API of each tool at regular intervals to check whether there are any new messages or files.

[0389] 3. The server extracts the following information from the retrieved message or file:

[0390] Message body

[0391] Attachments

[0392] timestamp (time of sending)

[0393] Sender information (email address, name, etc.)

[0394] 4. The server stores the extracted data in a database in JSON or other data formats.

[0395] Step 2: Analyze the data

[0396] 1. The server analyzes messages and attachments stored in a database using natural language processing algorithms.

[0397] Example: Use SpaCy or BERT to extract key keywords and context.

[0398] 2. The server automatically classifies each message or file based on the extracted keywords.

[0399] Example: Classify data related to Project X based on the keyword "Project X".

[0400] 3. The server adds metadata such as timestamps and sender information to the classified data.

[0401] Step 3: Perform a search

[0402] 1. The user accesses a dedicated search screen and enters a search term.

[0403] For example, enter "Project X Report."

[0404] 2. The server queries the database based on the entered search term.

[0405] Example: Search for "Project X Report" to get relevant data.

[0406] Step 4: Formatting and displaying search results

[0407] 1. The server sorts the retrieved search results by relevance.

[0408] Example: sorting search results based on relevance score.

[0409] 2. The server formats the search results in a format that is easy for the user to understand.

[0410] Each result includes the message body, attachments (with links), timestamps, and sender information.

[0411] 3. The user reviews the organized search results and accesses the information they need.

[0412] Through these steps, users can centrally manage information that is scattered across multiple communication tools and efficiently obtain the information they need.

[0413] Example 1

[0414] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0415] In today's world, it is common for people to use multiple information transmission tools for business and communication. However, it is difficult to centrally manage and analyze data from these information transmission tools and efficiently search and acquire necessary information. In particular, there is a demand for a system that can uniformly process data in different formats and quickly extract highly relevant information. The objective of this invention is to provide a system that solves these data management and analysis problems and enables users to efficiently acquire necessary information.

[0416] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0417] In this invention, the server includes means for collecting data from multiple information transmission tools, means for storing the collected data in a data management device, means for analyzing the stored data using a natural language processing algorithm, means for automatically classifying the analyzed data, means for adding metadata to the classified data, means for searching the data management device based on search terms entered by an information user on a search screen, means for displaying search results in order of relevance, means for using SpaCy or BERT as a natural language processing algorithm, means for storing the collected data in JSON format, and means for periodically checking for new messages and files from the information transmission tools. This enables information users to efficiently search and retrieve data centrally managed by different information transmission tools.

[0418] An "information communication tool" is a system for sending and receiving messages or files in digital form, such as email, chat applications, or online messaging systems.

[0419] A "data management device" is a database or storage system for storing, managing, and searching collected data.

[0420] "Natural language processing algorithms" are algorithms used to analyze text data, extract keywords, understand context, classify categories, etc. Examples include SpaCy and BERT.

[0421] "Metadata" refers to attribute information of data, and includes, for example, the date and time of data creation, sender information, related categories, and the like.

[0422] The "search screen" is a user interface that allows an information user to input a search word and execute a search query on the data management device.

[0423] "Relevance order" refers to the order in which search results are sorted in order of the most relevant to the information user's search word.

[0424] "SpaCy" is an open source Python library for implementing advanced natural language processing.

[0425] "BERT" is a transformer-based natural language processing model and an algorithm for performing highly accurate text analysis.

[0426] "JSON format" is an abbreviation for JavaScript Object Notation, and is a data format that represents structured data in a text format that is easy for humans to read and machine to process.

[0427] "Periodic check for new messages and files" refers to the process where the server calls the API at regular intervals to check for newly added messages and files.

[0428] MODE FOR CARRYING OUT THE INVENTION

[0429] The present invention provides a system that collects data from multiple information transmission tools, stores it in a data management device, analyzes the data using a natural language processing algorithm, and provides related information based on search terms. This system allows information users to efficiently and quickly obtain the information they need.

[0430] First, the server collects data from multiple communication tools, including email, chat applications, and online messaging systems. In this process, the server obtains API access using each tool's API key or OAuth token. For example, it periodically checks for new messages and files using the APIs of Gmail and chat applications to obtain data. The obtained data is then stored in a data management device in JSON format.

[0431] The server then analyzes the stored data using natural language processing algorithms. This analysis uses advanced natural language processing libraries such as SpaCy and BERT. The server tokenizes the text data and automatically classifies it by extracting keywords and performing contextual analysis. Key keywords and context are extracted from the content of messages and files, and they are then classified into categories such as "progress report" or "meeting minutes." Metadata such as timestamps and sender information is also added to the data.

[0432] The information user then uses a dedicated search screen to enter search terms to find the information they need. The device sends the search terms to the server, which then queries the data management device to retrieve related messages and files. The retrieved search results are sorted by relevance and displayed in a format that is easy for the information user to view. For example, if the information user searches for "Project X report," the server will display a list of reports and emails related to Project X, providing the message body, links to attachments, the time of sending, sender information, and more.

[0433] As a concrete example, let us consider the case where a server collects data from a chat application. The server first uses the chat application's API to obtain new messages. It extracts the content, attachments, timestamps, and sender information from the obtained messages and stores them in a data management device. The server then analyzes this data and classifies it into categories such as "progress reports" and "meeting minutes." When an information user searches for "meeting minutes," messages and files related to that search term are displayed. The server displays the search results in chronological order and by importance, allowing the information user to quickly access the information they need.

[0434] Examples of prompts for generative AI models include:

[0435] How can I get unread emails using the Email API?

[0436] "Explain how to extract keywords from text using a natural language processing library."

[0437] "What's the best way to construct a search query?"

[0438] In this way, by using this system, information users can efficiently search and retrieve data that is centrally managed across multiple information transmission tools, significantly improving work efficiency.

[0439] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0440] Step 1: Collect data

[0441] The server establishes API access. Specifically, it obtains API access rights using the API key or OAuth token of an information transmission tool such as Gmail or a chat application.

[0442] Input: API key or OAuth token of the information transmission tool

[0443] How it works: Obtain an access token through the API authentication process

[0444] Output: Access token

[0445] The server acquires data by collecting new messages and files from each information transmission tool via API.

[0446] Input: Access token, API endpoint

[0447] What it does: For example, call an endpoint that uses the Gmail API to get unread emails.

[0448] Output: Collected data (message body, attachments, timestamp, sender information)

[0449] The server saves the collected data in a data management device. The collected data is stored in the data management device in JSON format.

[0450] Input: Collected data (JSON format)

[0451] Action: Executes an INSERT statement to the database and saves the data.

[0452] Output: Data stored in the data management device

[0453] Step 2: Analyze the data

[0454] The server applies natural language processing algorithms, using libraries such as SpaCy and BERT, to analyze the stored data.

[0455] Input: Text data stored in the data management device

[0456] Operation: Tokenizes text data, extracts keywords, and performs contextual analysis

[0457] Output: Analysis results (extracted keywords and context information)

[0458] The server then categorizes the data based on the analysis results, categorizing messages and files based on extracted keywords and context.

[0459] Input: Analysis results

[0460] Behavior: For example, classifying data into categories such as "progress reports" and "meeting minutes"

[0461] Output: Classified data

[0462] The server adds metadata, such as a timestamp and sender information, to the data.

[0463] Input: Classified data, timestamp information, caller information

[0464] Behavior: Adds metadata to each data entry

[0465] Output: Data with metadata

[0466] Step 3: Search and display in the user interface

[0467] Users use a dedicated search screen to input search terms to find the information they need.

[0468] Input: Search word

[0469] Operation: The device sends the search word to the server.

[0470] Output: Search request to the server

[0471] The server performs a database query, querying the data management device based on the entered search words.

[0472] Input: Search word

[0473] Operation: Searches for the relevant data using the SQL SELECT statement

[0474] Output: Query results (associated messages and files)

[0475] The server formats the search results in order of relevance, making the retrieved data easy for information users to view.

[0476] Input: Query results

[0477] What it does: Sorts search results by relevance and formats them in HTML.

[0478] Output: Formatted search results

[0479] The terminal displays the search results, ie, displays the search results returned from the server to the user.

[0480] Input: Formatted search results

[0481] Action: Show search results

[0482] Output: Search results displayed to the user

[0483] In this way, by processing each step sequentially, this system enables information users to efficiently search and retrieve data that is centrally managed across multiple information transmission tools.

[0484] (Application example 1)

[0485] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0486] Previously, maintenance information for machinery and equipment within a factory was managed separately using multiple communication methods, making centralized management of the information difficult. Furthermore, searching and analyzing maintenance information required a great deal of time and effort, making efficient maintenance management difficult. This resulted in delays in responding to machine breakdowns and malfunctions, leading to reduced production efficiency.

[0487] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0488] In this invention, the server includes means for collecting data from multiple communication means, means for storing the collected data in an information storage device, and means for analyzing the stored data using a natural language processing algorithm, thereby enabling the centralized collection and analysis of information related to the maintenance of factory equipment and the prompt provision of necessary information.

[0489] "Multiple means of communication" refers to a variety of digital communication tools such as email, chat, and maintenance logs.

[0490] "Data collection method" refers to the function of periodically obtaining new messages and files using APIs and OAuth tokens of multiple communication methods.

[0491] "Information storage device" refers to a digital storage medium such as a database or cloud storage for storing collected data.

[0492] "Natural language processing algorithm" refers to technology that analyzes text data using methods such as SpaCy and BERT and extracts keywords and contextual information.

[0493] "Automatic classification means" refers to a function that automatically classifies analyzed data based on specific criteria or categories.

[0494] "Attribute information" refers to metadata such as timestamps and sender information that is added to data.

[0495] "Search instructions" refer to keywords or phrases that a user enters into a search screen.

[0496] "Search means" refers to a function that queries an information storage device based on a search instruction entered by a user to retrieve relevant data.

[0497] "Means for displaying in order of relevance" refers to a function for sorting search results in order of relevance and displaying them to the user.

[0498] "Information on the maintenance of factory equipment" refers to data related to the maintenance, inspection, and repair of machinery and equipment within a factory.

[0499] The present invention relates to a system for efficiently managing information relating to the maintenance of factory equipment and for quickly providing necessary information. Specific embodiments of the present invention will be described below.

[0500] First, the server collects data from multiple communication methods required within the factory (e.g., email, chat, maintenance logs). The data is acquired using the API of each communication method and stored in an information storage device (database or cloud storage).

[0501] The server then analyzes the stored data using natural language processing algorithms. Specifically, it uses natural language processing libraries such as SpaCy and BERT to analyze the text data and extract keywords and contextual information. This allows for automatic classification of the data and the addition of attribute information (such as timestamps and sender information).

[0502] Users input search instructions into a search screen using a display or tablet installed on a factory robot. Based on this, the server queries the information storage device and retrieves relevant data. Search results are displayed in order of relevance and presented in a format that is intuitively easy for users to understand.

[0503] For example, if a factory robot searches for "Machine A's maintenance history for the past six months," the system quickly displays detailed historical information, including timestamps and maintenance details. Similarly, if a user searches for "error messages," the system also displays related repair procedures and contact information.

[0504] This system centrally manages information related to factory equipment maintenance and provides necessary information quickly, which speeds up responses to machine breakdowns and malfunctions and improves production efficiency.

[0505] Examples of prompts for a generative AI model might include:

[0506] "We want to develop a system that collects maintenance log data, analyzes its contents, and provides maintenance history and error information. What architecture and technology would be suitable?"

[0507] In this manner, specific elements and procedures are described in detail to clearly provide specific embodiments of the invention.

[0508] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0509] Step 1:

[0510] The server collects data from multiple communication methods within the factory (email, chat, maintenance logs). Specifically, it uses APIs and OAuth tokens to obtain access to each communication method and periodically checks whether there are any new messages or files. The input requires the API key or OAuth token for each communication method, and the output is the collected message and file data.

[0511] Step 2:

[0512] The server stores the collected data in an information storage device (database). Specifically, it extracts attribute information such as the contents of the collected messages and files, timestamps, and sender information, converts it into an appropriate format (e.g., JSON format), and stores it in the database. The input is the data collected in step 1, and the output is the data stored in the information storage device.

[0513] Step 3:

[0514] The server analyzes the data stored in the information storage device using natural language processing algorithms (e.g., SpaCy or BERT). Specifically, it analyzes the text data to extract key keywords and contextual information, and then classifies the data based on that. The input is the unanalyzed data stored in the information storage device, and the output is the analyzed data and its keyword information.

[0515] Step 4:

[0516] The server adds attribute information, such as a timestamp and sender information, to the analyzed data. This allows the analyzed data to be classified in more detail and supplemented with related information. The input is the analyzed data, and the output is the analyzed data with the attribute information added.

[0517] Step 5:

[0518] The user inputs search instructions using a tablet or display installed on the factory robot. Specifically, they input keywords or phrases (e.g., "Maintenance history of machine A") into the search screen. The input is the search instruction entered by the user, and the output is the search instruction itself.

[0519] Step 6:

[0520] The server queries the information storage device based on the search instructions entered by the user. Specifically, it finds data corresponding to the search keywords and retrieves related data. The input is the search instructions entered by the user, and the output is the related data as the search results.

[0521] Step 7:

[0522] The server sorts the retrieved search results in order of relevance and provides them to the user. Specifically, it formats the search results in order of relevance and displays them on the display in a format that is intuitively easy for the user to understand. The input is related data as search results, and the output is a display of the formatted search results.

[0523] Step 8:

[0524] The user selects the required information from the displayed search results and checks the details. Specifically, the user clicks on the relevant information from the list of search results to open the details screen. The input is the list of search results, and the output is the selected detailed information.

[0525] The above steps will realize a system that efficiently collects and manages information related to factory equipment maintenance and quickly provides the necessary information. This system will enable a quick response to machine breakdowns and malfunctions, contributing to improved production efficiency.

[0526] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0527] MODE FOR CARRYING OUT THE INVENTION

[0528] This invention is a system that combines a system that collects data from multiple communication tools, stores the data in a database, and analyzes and classifies it using natural language processing algorithms with an emotion engine that recognizes the user's emotions. Specific embodiments of this system and the program processing procedures are described below.

[0529] 1. Data collection

[0530] The server obtains access using API keys or OAuth tokens from Gmail, GChat, or other communication tools. The server uses these APIs to periodically check for new messages or files. For example, to collect data from Gmail, the server uses the Gmail API to retrieve unread emails and extract the email content, attachments, sent time (timestamp), and sender information. This extracted data is then stored in a database in a data format such as JSON.

[0531] 2. Data Analysis

[0532] The server then analyzes the stored data using natural language processing algorithms (NLP), such as SpaCy or BERT, to extract key keywords and context from the content of message bodies and attachments. This analysis identifies criteria and categories for classifying the data. The server then automatically classifies each message or file based on the extracted keywords and context. Furthermore, the server complements the data with metadata such as timestamps and sender information.

[0533] 3. User Emotion Recognition

[0534] The server uses an emotion engine to recognize the user's emotions when the user accesses the search screen and enters a search term. The emotion engine evaluates the user's emotional state through facial recognition, voice analysis, and text analysis. For example, when a user searches for "Project X report," the emotion engine analyzes the user's voice tone, facial expression, and the tone of the text entered to understand the user's current emotional state.

[0535] 4. Tailor search results based on sentiment

[0536] The server adjusts the order of search results based on the user's emotions, as determined by the emotion engine. For example, if the user is feeling stressed, the server prioritizes relevant results and reduces unnecessary information. If the user is feeling relaxed, the server also displays detailed explanations and related materials.

[0537] 5. Search Results Display and Optimization

[0538] Users can check the search results provided by the server. The server continuously analyzes the user's reactions using an emotion engine and optimizes the display of search results from the next time. For example, if a user has a positive reaction to a particular search result, the server will display results based on that reaction when a similar search is performed.

[0539] Specific examples

[0540] For example, if a user searches for "Project X meeting minutes," the server will search for meeting minutes messages related to Project X and display them in order of relevance along with metadata such as timestamps and sender information. At this time, the emotion engine will adjust the display order of search results as needed based on the user's emotional state. The emotion engine also determines whether the user is satisfied with the search results, which will be used to improve the accuracy of search results in future searches.

[0541] This system allows users to centrally manage what information has been exchanged with which tools, enabling them to efficiently obtain the information they need. In addition, by combining it with an emotion engine, it is possible to display information optimally according to the user's emotional state, further improving work efficiency.

[0542] The processing flow will be explained below.

[0543] Step 1: Collect data

[0544] 1. The server obtains access rights using the API key or OAuth token of each communication tool (e.g., Gmail, GChat).

[0545] 2. The server calls the API of each tool at regular intervals to check whether there are any new messages or files.

[0546] 3. The server extracts the following information from the retrieved message or file:

[0547] Message body

[0548] Attachments

[0549] timestamp (time of sending)

[0550] Sender information (email address, name, etc.)

[0551] 4. The server stores the extracted data in a database in JSON or other data formats.

[0552] Step 2: Analyze the data

[0553] 1. The server analyzes messages and attachments stored in a database using natural language processing algorithms.

[0554] For example, use SpaCy or BERT to extract key keywords and context.

[0555] 2. The server automatically classifies each message or file based on the extracted keywords.

[0556] For example, data related to Project X may be classified based on the keyword "Project X."

[0557] 3. The server adds metadata such as timestamps and sender information to the classified data.

[0558] Step 3: Acquire data for emotion recognition

[0559] 1. The user accesses a dedicated search screen and prepares to enter a search term.

[0560] 2. The device collects the user's facial expression and voice data when the user enters a search term.

[0561] For example, a webcam and microphone are used to capture the user's facial expressions and voice tone.

[0562] 3. The server analyzes the collected emotion data using an emotion engine.

[0563] For example, facial recognition software and voice analysis tools are used to determine a user's emotional state.

[0564] Step 4: Perform sentiment-based search

[0565] 1. The user enters a search term into a dedicated search screen.

[0566] For example, enter "Project X Report."

[0567] 2. The server queries the database using the emotion data from the emotion engine along with the entered search term.

[0568] For example, if you search for "Project X report" and the user is feeling stressed, the system will prioritize the most relevant results.

[0569] Step 5: Formatting and displaying search results

[0570] 1. The server sorts the retrieved search results by relevance.

[0571] For example, the analysis results from the emotion engine are taken into account to display more relevant results at the top.

[0572] 2. The server formats the search results in a format that is easy for the user to understand.

[0573] Each result includes the message body, attachments (with links), timestamps, and sender information.

[0574] 3. The user reviews the organized search results and accesses the information they need.

[0575] Step 6: Collect and optimize responses to search results

[0576] 1. The device collects the user's reactions as they check the search results.

[0577] For example, it monitors the user's facial expressions, voice tone, clicking behavior, etc.

[0578] 2. The server analyzes the reaction data with an emotion engine to evaluate how users reacted to a particular search result.

[0579] 3. The server will optimize the search results displayed from the next time onwards based on the analysis results.

[0580] For example, the search algorithm can be adjusted to prioritize results that satisfy users.

[0581] This allows the user to efficiently obtain the necessary information and enables optimal information display according to the user's emotional state.

[0582] Example 2

[0583] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0584] In today's business environment, a large amount of information is exchanged through multiple communication tools, creating a need for centralized data management and efficient information search. However, due to a lack of optimization of search results that take into account the user's emotional state, it is difficult to provide appropriate information according to the user's state. This leads to problems such as reduced work efficiency and increased stress for users.

[0585] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0586] In this invention, the server includes means for collecting data from multiple communication tools, means for saving the collected data in a database, means for analyzing the saved data using a natural language processing algorithm, means for automatically classifying the analyzed data, means for adding metadata to the classified data, means for searching the database based on search words entered by a user on a search screen, means for displaying search results in order of relevance, means for using an emotion engine that recognizes the user's emotions when the user accesses the search screen, means for adjusting the display order of search results based on the emotion information recognized by the emotion engine, and means for continuously analyzing the user's reactions using the emotion engine and optimizing the display of search results from the next time onwards. This enables centralized management of information and provision of appropriate information according to the user's emotional state.

[0587] "Multiple communication tools" refers to various means by which users exchange messages and data electronically, such as email and chat applications.

[0588] "Means for collecting data" refers to software and hardware components used to obtain message and file information from designated communication tools.

[0589] "Means for storing data in a database" refers to a storage device or management system for permanently storing collected data.

[0590] "Natural language processing algorithms" refer to machine learning and language models that analyze collected text data and extract context and meaning.

[0591] "Means for automatically classifying data" refers to a function that divides data into specific categories or tags based on the results of analysis.

[0592] "Metadata" refers to additional information about a message or file (e.g., timestamps and originator information).

[0593] A "search screen" refers to an interface that allows a user to input search keywords and search for information in a database.

[0594] "Emotion engine" refers to technology that performs facial recognition, voice analysis, and text analysis of a user to assess their emotional state.

[0595] "Means for adjusting the display order of search results" refers to a function for sorting search results in an appropriate order based on the user's emotional state.

[0596] "Means for continuously analyzing user responses" refers to technology that collects user search activity and responses to the results, and optimizes the display of search results from the next time onwards.

[0597] MODE FOR CARRYING OUT THE INVENTION

[0598] This invention is a system that combines a system that collects data from multiple communication tools, stores the data in a database, and analyzes and classifies the data using a natural language processing algorithm with an emotion engine that recognizes the user's emotions. Specific embodiments for implementing this system will be described below.

[0599] 1. Data collection

[0600] The server obtains access rights from multiple communication tools, such as Gmail and chat tools, using API keys and OAuth tokens. It periodically checks for new messages and files using the Gmail API and chat API. For example, when collecting data from Gmail, the server retrieves unread emails and extracts the email content, attachments, sent date and time, and sender information. This extracted data is stored in a database in a data format such as JSON.

[0601] 2. Data Analysis

[0602] The server then analyzes the stored data using natural language processing algorithms, particularly tools like SpaCy and BERT. These tools extract key keywords and context from the content of messages and attachments. This analysis determines the criteria and categories for categorizing the data. The server automatically classifies each message or file based on the extracted keywords and context. It also adds metadata such as timestamps and sender information to complement the data.

[0603] 3. User Emotion Recognition

[0604] When a user accesses the search screen from their device and enters a search term, the server uses an emotion engine to recognize the user's emotion. The emotion engine evaluates the user's emotional state through facial recognition, voice analysis, and text analysis. For example, when a user searches for "Project X Report," the server analyzes the user's voice tone, facial expression, and the tone of the text entered to understand the user's current emotional state.

[0605] 4. Tailor search results based on sentiment

[0606] The server adjusts the display order of search results based on the emotional information recognized by the emotion engine. For example, if the user is feeling stressed, the server will prioritize relevant results and reduce unnecessary information. If the user is feeling relaxed, the server will also display detailed explanations and related materials.

[0607] 5. Search Results Display and Optimization

[0608] The server displays the adjusted search results on the user's device. The user can then review the displayed search results. Furthermore, the server continuously analyzes the user's reactions using an emotion engine to optimize the display of search results from the next time onwards. For example, if a user has a positive reaction to a particular search result, the server will display results based on that reaction when a similar search is performed.

[0609] Examples of concrete examples and prompts

[0610] For example, consider the case where a user searches for "Project X meeting minutes." The server searches the database for messages of meeting minutes related to Project X and sorts them in order of relevance along with metadata such as timestamps and sender information. The emotion engine then adjusts the display order of the search results as needed based on the user's emotional state. The emotion engine also analyzes the user's reaction to the search results and optimizes the display of search results from the next time onwards.

[0611] An example of a prompt sentence might be "Please search for Project X meeting minutes."

[0612] This system allows users to centrally manage what information has been exchanged using which tools, enabling them to efficiently obtain the information they need. In addition, the introduction of an emotion engine makes it possible to provide optimal information according to the user's emotional state, significantly improving work efficiency.

[0613] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0614] Step 1: Collect data

[0615] 1.1 The server obtains API keys and OAuth tokens from multiple communication tools such as Gmail and chat tools. The input is the authentication information of each tool, and the output is the obtained access rights information.

[0616] 1.2 The server uses the obtained access right information to access each tool and periodically checks whether new messages or files exist. The input is the access right information, and the output is the new message or file information.

[0617] 1.3 The server retrieves unread emails using the Gmail API and extracts the email content, attachments, sent date and time, and sender information. The input is access rights information and new message information, and the output is the extracted email data.

[0618] 1.4 The server saves the extracted data in a database in JSON format. The input is the extracted email data, and the output is the data saved in the database.

[0619] Step 2: Analyze the data

[0620] 2.1 The server analyzes the stored data based on a natural language processing algorithm, where the input is the stored data and the output is the analysis result.

[0621] 2.2 The server uses natural language processing tools such as SpaCy or BERT to extract key keywords and context from the content of the message body and attachments. The input is the stored data, and the output is the extracted keywords and context.

[0622] 2.3 The server automatically classifies each message or file based on the extracted keywords and context. The input is the extracted keywords and context, and the output is the classified data.

[0623] 2.4 The server complements the data by adding metadata such as timestamps and sender information. The input is the classified data, and the output is the data with the metadata added.

[0624] Step 3: Recognizing user emotions

[0625] 3.1 The user accesses the search screen from their device and enters a search term. The input is the search term, and the output is the transmission of the search term.

[0626] 3.2 The server passes the user's input to the emotion engine. The input is the search word, and the output is the input to the emotion engine.

[0627] 3.3 The emotion engine evaluates the user's emotional state through facial recognition, voice analysis, and text analysis. The input is the search term and the user's biometric information, and the output is the user's emotional state.

[0628] 3.4 For example, when a user searches for "Project X report," the emotion engine analyzes the user's tone of voice, facial expression, and the tone of the text entered to understand the user's current emotional state. The input is the search word and the user's biometric information, and the output is the analyzed emotional state.

[0629] Step 4: Tailor search results based on sentiment

[0630] 4.1 The server adjusts the display order of search results based on the user's emotional information recognized by the emotion engine. The input is the user's emotional information, and the output is the adjusted search results.

[0631] 4.2 For example, if a user is feeling stressed, the server prioritizes relevant results and reduces unnecessary information. The input is the user's emotional information, and the output is the prioritized search results.

[0632] 4.3 If the user is relaxed, detailed explanations and related materials are also displayed. The input is the user's emotional information, and the output is search results with added detailed information.

[0633] Step 5: Displaying and optimizing search results

[0634] 5.1 The server displays the tailored search results on the user's terminal. The input is the tailored search results and the output is the results displayed on the terminal.

[0635] 5.2 The user can confirm the search results displayed on the terminal. The input is the displayed search results, and the output is the user's confirmation.

[0636] 5.3 The server analyzes user responses to search results (clicks, time spent, etc.) using an emotion engine. The input is the user response, and the output is the analysis result.

[0637] 5.4 The server will optimize the display of search results from the next time onwards based on the analysis results. The input is the analysis results, and the output is the optimized search result display format.

[0638] This series of processing steps allows the system to provide optimal information according to the user's emotional state, allowing the user to efficiently obtain the information they need and improve their work efficiency.

[0639] (Application example 2)

[0640] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0641] In recent years, brick-and-mortar stores have been required to provide services that respond promptly to customer needs, but it is difficult to grasp the emotional state of customers in real time, making it difficult to provide optimal customer service and product recommendations. Furthermore, there is a demand for methods to improve customer satisfaction by utilizing past communication data, but there is a lack of efficient ways to do this. There is an urgent need to provide a system that improves customer experience by resolving these issues.

[0642] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0643] In this invention, the server includes means for collecting data from multiple communication tools, means for saving the collected data in a database, means for analyzing the saved data using a natural language processing algorithm, means for automatically classifying the analyzed data, means for adding metadata to the classified data, means for searching the database based on search words entered by a user on a search screen, means for displaying search results in order of relevance, means for recognizing the user's emotions in real time, and means for adjusting the display order of information based on the recognized emotions. This makes it possible to grasp the emotional state of customers in real time and provide optimal customer service and product suggestions based on past communication data.

[0644] 1. "Communication tools" is a general term for software and hardware used to send and receive information via digital media, such as email and chat applications.

[0645] 2. "Database" means a data management system that efficiently manages collected data and allows for searching and analysis as needed.

[0646] 3. "Natural language processing algorithm" is a general term for algorithms that analyze text data and understand and process its meaning.

[0647] 4. "Classification" is the process of grouping analyzed data based on specific criteria.

[0648] 5. "Metadata" is a general term for attribute information and additional information related to data, which is added to facilitate data search and management.

[0649] 6. "Search screen" is an interface that allows users to enter keywords and search for information in the database.

[0650] 7. "Relevance" is a criterion for sorting search results based on the degree of match or importance with the keywords entered by the user.

[0651] 8. "Emotion recognition means" is a general term for technologies and devices that analyze a user's facial expressions and vocal tone to understand their emotional state.

[0652] 9. "Adjusting the information display order" is the process of changing the order of displayed information based on perceived user sentiment.

[0653] This invention is a system for grasping the emotional state of customers in real time in a physical store and providing optimal customer service and product recommendations. This system collects data from multiple communication tools and analyzes it with a natural language processing algorithm to grasp the user's emotional state in real time and adjust the order in which information is displayed based on this data.

[0654] First, the server collects data from communication tools such as Gmail and chat apps. The server uses the API keys or OAuth tokens of these tools to gain access and periodically checks for new messages and files. The collected data is stored in a database in a data format such as JSON, allowing salespeople to quickly reference past interactions with customers.

[0655] The server then analyzes the stored data using natural language processing algorithms. Generative AI models like SpaCy and BERT are used to extract key keywords and context from the content of messages and attachments. This analysis identifies criteria or categories for classifying the data. Classification tags and metadata are then added to the analyzed data.

[0656] Additionally, the system employs emotion recognition to recognize users' emotions in real time. This is achieved by using the camera and microphone built into the smart glasses to analyze the customer's facial expressions and tone of voice. For emotion recognition, the system uses the Hugging Face Transformers library to evaluate their emotional state. This emotion data is stored in the cloud and can be used to trigger specific actions.

[0657] The system also provides a means to adjust the order in which information is displayed based on emotions. For example, if a customer is feeling stressed, relevant search results will be displayed first, reducing unnecessary information. On the other hand, if the customer is relaxed, detailed explanations and related materials will also be displayed. This adjustment makes it possible to provide the optimal service according to the customer's state.

[0658] The following specific examples are given as examples.

[0659] Example 1: When a customer searches for "Project X meeting minutes"

[0660] The server searches for meeting minutes related to Project X and displays them in order of relevance along with timestamps and sender information. The emotion engine adjusts the display order according to the user's emotional state and provides detailed information as needed.

[0661] Example 2: When a customer is stressed

[0662] Prompt: "What is the appropriate way to respond when a customer is stressed?"

[0663] Example output: "Respond simply and quickly, and refrain from suggesting new products."

[0664] Example 3: When the customer is relaxed

[0665] Prompt: "What is the appropriate way to respond when a customer is relaxed?"

[0666] Sample output: "Provides detailed product descriptions and offers additional options."

[0667] This will enable customer service in physical stores to respond more quickly to the emotional state of customers and provide greater satisfaction.

[0668] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0669] Step 1:

[0670] The server obtains access rights using API keys or OAuth tokens from communication tools such as Gmail and chat apps. The input is the authentication information for each communication tool, and the output is the access rights to these tools. Specifically, the server periodically checks for and collects unread emails using the Gmail API.

[0671] Step 2:

[0672] The server collects new messages and files from the communication tools it has gained access to and stores them in a database in an appropriate format, such as JSON. The input is the data of the new messages and files, and the output is the data stored in the database. Specifically, timestamps and sender information are also stored in the database at the same time.

[0673] Step 3:

[0674] The server analyzes the stored data using natural language processing algorithms (e.g., SpaCy or BERT). The input is the text data stored in the database, and the output is the analysis results, including key keywords and contextual information. Specifically, meaning is extracted from the contents of the message body and attachments and automatically classified.

[0675] Step 4:

[0676] The server adds classification tags and metadata to the analyzed data. The input is the main keywords and context information from the analysis results, and the output is data with the classification tags and metadata added. Specifically, category information, timestamp, and sender information are added to each data.

[0677] Step 5:

[0678] A user enters keywords into a search screen, and the server searches the database based on those keywords. The input is the search keyword entered by the user, and the output is a list of related data. Specifically, the server extracts highly relevant data from the searched data.

[0679] Step 6:

[0680] The server sorts the search results in order of relevance and displays them to the user. The input is a list of searched data, and the output is the search results sorted in order of relevance. Specifically, the list is displayed taking into account related metadata.

[0681] Step 7:

[0682] The system grasps the user's emotional state in real time through a camera and microphone connected to a device (such as smart glasses). The input is camera video and microphone audio, and the output is the analysis result of the user's emotional state. Specifically, it uses the Hugging Face Transformers library to recognize emotions from facial expressions and voice tone.

[0683] Step 8:

[0684] The server adjusts the display order of information based on the recognized emotional state. The input is the analysis result of the user's emotional state and the search results, and the output is the adjusted display order of information. Specifically, if the user is feeling stressed, highly relevant information is displayed first, and if the user is relaxed, additional detailed information is displayed.

[0685] Step 9:

[0686] The server continuously analyzes customer responses using an emotion engine to optimize the display of search results from the next time onwards. The input is customer response data, and the output is search results in an optimized display order. Specifically, the next display is improved based on search results that have generated positive responses.

[0687] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0688] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0689] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0690] [Third embodiment]

[0691] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0692] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0693] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0694] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0695] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0696] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0697] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0698] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0699] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0700] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0701] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0702] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0703] MODE FOR CARRYING OUT THE INVENTION

[0704] This invention is a system that collects data from multiple communication tools, stores it in a database, analyzes the data using a natural language processing algorithm, and provides related information based on search terms. Specific embodiments of this system and the program processing procedures are described below.

[0705] 1. Data collection

[0706] First, the server gets access to Gmail, GChat, and other communication tools using their API keys or OAuth tokens, and then periodically checks for new messages and files.

[0707] For example, if a server collects data from Gmail, it uses the Gmail API to retrieve unread emails and extracts the email content, attachments, sent time (timestamp), and sender information. This extracted data is then stored in a database in a data format such as JSON.

[0708] 2. Data Analysis

[0709] The server then analyzes the stored data using natural language processing algorithms (NLP), such as SpaCy or BERT, to extract key keywords and context from the content of the message body and attachments. This analysis identifies criteria and categories for classifying the data.

[0710] The server automatically categorizes each message or file based on extracted keywords and context, and complements the data with metadata such as timestamps and sender information.

[0711] 3. Search and display in the user interface

[0712] Users use a dedicated search screen to enter search terms to find the information they need. The server queries the database based on the search terms and retrieves related messages and files. The retrieved search results are sorted by relevance and displayed in an easy-to-read format for the user.

[0713] For example, if a user searches for "Project X report," the server will search for report messages related to Project X and display them in order of relevance, along with metadata such as timestamps and sender information. The search results include the message body, links to attachments, time of sending, and sender information.

[0714] Specific examples

[0715] As a concrete example, let's consider the case where a server collects data from G Chat. In this case, the server first obtains new messages using the G Chat API. The server then extracts the contents, attachments, timestamps, sender information, etc. from the obtained messages and stores them in a database.

[0716] The server then analyzes this data and categorizes it into categories, such as "progress reports" or "meeting minutes." If a user searches for "meeting minutes," messages and files related to that search term are displayed. The server displays results based on chronology and importance, allowing users to quickly access the information they need.

[0717] This system allows users to centrally manage what information has been exchanged with which tools, and efficiently obtain the information they need, greatly improving work efficiency.

[0718] The processing flow will be explained below.

[0719] Step 1: Collect data

[0720] 1. The server obtains access rights using the API key or OAuth token of each communication tool (e.g., Gmail, GChat).

[0721] 2. The server calls the API of each tool at regular intervals to check whether there are any new messages or files.

[0722] 3. The server extracts the following information from the retrieved message or file:

[0723] Message body

[0724] Attachments

[0725] timestamp (time of sending)

[0726] Sender information (email address, name, etc.)

[0727] 4. The server stores the extracted data in a database in JSON or other data formats.

[0728] Step 2: Analyze the data

[0729] 1. The server analyzes messages and attachments stored in a database using natural language processing algorithms.

[0730] Example: Use SpaCy or BERT to extract key keywords and context.

[0731] 2. The server automatically classifies each message or file based on the extracted keywords.

[0732] Example: Classify data related to Project X based on the keyword "Project X".

[0733] 3. The server adds metadata such as timestamps and sender information to the classified data.

[0734] Step 3: Perform a search

[0735] 1. The user accesses a dedicated search screen and enters a search term.

[0736] For example, enter "Project X Report."

[0737] 2. The server queries the database based on the entered search term.

[0738] Example: Search for "Project X Report" to get relevant data.

[0739] Step 4: Formatting and displaying search results

[0740] 1. The server sorts the retrieved search results by relevance.

[0741] Example: sorting search results based on relevance score.

[0742] 2. The server formats the search results in a format that is easy for the user to understand.

[0743] Each result includes the message body, attachments (with links), timestamps, and sender information.

[0744] 3. The user reviews the organized search results and accesses the information they need.

[0745] Through these steps, users can centrally manage information that is scattered across multiple communication tools and efficiently obtain the information they need.

[0746] Example 1

[0747] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0748] In today's world, it is common for people to use multiple information transmission tools for business and communication. However, it is difficult to centrally manage and analyze data from these information transmission tools and efficiently search and acquire necessary information. In particular, there is a demand for a system that can uniformly process data in different formats and quickly extract highly relevant information. The objective of this invention is to provide a system that solves these data management and analysis problems and enables users to efficiently acquire necessary information.

[0749] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0750] In this invention, the server includes means for collecting data from multiple information transmission tools, means for storing the collected data in a data management device, means for analyzing the stored data using a natural language processing algorithm, means for automatically classifying the analyzed data, means for adding metadata to the classified data, means for searching the data management device based on search terms entered by an information user on a search screen, means for displaying search results in order of relevance, means for using SpaCy or BERT as a natural language processing algorithm, means for storing the collected data in JSON format, and means for periodically checking for new messages and files from the information transmission tools. This enables information users to efficiently search and retrieve data centrally managed by different information transmission tools.

[0751] An "information communication tool" is a system for sending and receiving messages or files in digital form, such as email, chat applications, or online messaging systems.

[0752] A "data management device" is a database or storage system for storing, managing, and searching collected data.

[0753] "Natural language processing algorithms" are algorithms used to analyze text data, extract keywords, understand context, classify categories, etc. Examples include SpaCy and BERT.

[0754] "Metadata" refers to attribute information of data, and includes, for example, the date and time of data creation, sender information, related categories, and the like.

[0755] The "search screen" is a user interface that allows an information user to input a search word and execute a search query on the data management device.

[0756] "Relevance order" refers to the order in which search results are sorted in order of the most relevant to the information user's search word.

[0757] "SpaCy" is an open source Python library for implementing advanced natural language processing.

[0758] "BERT" is a transformer-based natural language processing model and an algorithm for performing highly accurate text analysis.

[0759] "JSON format" is an abbreviation for JavaScript Object Notation, and is a data format that represents structured data in a text format that is easy for humans to read and machine to process.

[0760] "Periodic check for new messages and files" refers to the process where the server calls the API at regular intervals to check for newly added messages and files.

[0761] MODE FOR CARRYING OUT THE INVENTION

[0762] The present invention provides a system that collects data from multiple information transmission tools, stores it in a data management device, analyzes the data using a natural language processing algorithm, and provides related information based on search terms. This system allows information users to efficiently and quickly obtain the information they need.

[0763] First, the server collects data from multiple communication tools, including email, chat applications, and online messaging systems. In this process, the server obtains API access using each tool's API key or OAuth token. For example, it periodically checks for new messages and files using the APIs of Gmail and chat applications to obtain data. The obtained data is then stored in a data management device in JSON format.

[0764] The server then analyzes the stored data using natural language processing algorithms. This analysis uses advanced natural language processing libraries such as SpaCy and BERT. The server tokenizes the text data and automatically classifies it by extracting keywords and performing contextual analysis. Key keywords and context are extracted from the content of messages and files, and they are then classified into categories such as "progress report" or "meeting minutes." Metadata such as timestamps and sender information is also added to the data.

[0765] The information user then uses a dedicated search screen to enter search terms to find the information they need. The device sends the search terms to the server, which then queries the data management device to retrieve related messages and files. The retrieved search results are sorted by relevance and displayed in a format that is easy for the information user to view. For example, if the information user searches for "Project X report," the server will display a list of reports and emails related to Project X, providing the message body, links to attachments, the time of sending, sender information, and more.

[0766] As a concrete example, let us consider the case where a server collects data from a chat application. The server first uses the chat application's API to obtain new messages. It extracts the content, attachments, timestamps, and sender information from the obtained messages and stores them in a data management device. The server then analyzes this data and classifies it into categories such as "progress reports" and "meeting minutes." When an information user searches for "meeting minutes," messages and files related to that search term are displayed. The server displays the search results in chronological order and by importance, allowing the information user to quickly access the information they need.

[0767] Examples of prompts for generative AI models include:

[0768] How can I get unread emails using the Email API?

[0769] "Explain how to extract keywords from text using a natural language processing library."

[0770] "What's the best way to construct a search query?"

[0771] In this way, by using this system, information users can efficiently search and retrieve data that is centrally managed across multiple information transmission tools, significantly improving work efficiency.

[0772] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0773] Step 1: Collect data

[0774] The server establishes API access. Specifically, it obtains API access rights using the API key or OAuth token of an information transmission tool such as Gmail or a chat application.

[0775] Input: API key or OAuth token of the information transmission tool

[0776] How it works: Obtain an access token through the API authentication process

[0777] Output: Access token

[0778] The server acquires data by collecting new messages and files from each information transmission tool via API.

[0779] Input: Access token, API endpoint

[0780] What it does: For example, call an endpoint that uses the Gmail API to get unread emails.

[0781] Output: Collected data (message body, attachments, timestamp, sender information)

[0782] The server saves the collected data in a data management device. The collected data is stored in the data management device in JSON format.

[0783] Input: Collected data (JSON format)

[0784] Action: Executes an INSERT statement to the database and saves the data.

[0785] Output: Data stored in the data management device

[0786] Step 2: Analyze the data

[0787] The server applies natural language processing algorithms, using libraries such as SpaCy and BERT, to analyze the stored data.

[0788] Input: Text data stored in the data management device

[0789] Operation: Tokenizes text data, extracts keywords, and performs contextual analysis

[0790] Output: Analysis results (extracted keywords and context information)

[0791] The server then categorizes the data based on the analysis results, categorizing messages and files based on extracted keywords and context.

[0792] Input: Analysis results

[0793] Behavior: For example, classifying data into categories such as "progress reports" and "meeting minutes"

[0794] Output: Classified data

[0795] The server adds metadata, such as a timestamp and sender information, to the data.

[0796] Input: Classified data, timestamp information, caller information

[0797] Behavior: Adds metadata to each data entry

[0798] Output: Data with metadata

[0799] Step 3: Search and display in the user interface

[0800] Users use a dedicated search screen to input search terms to find the information they need.

[0801] Input: Search word

[0802] Operation: The device sends the search word to the server.

[0803] Output: Search request to the server

[0804] The server performs a database query, querying the data management device based on the entered search words.

[0805] Input: Search word

[0806] Operation: Searches for the relevant data using the SQL SELECT statement

[0807] Output: Query results (associated messages and files)

[0808] The server formats the search results in order of relevance, making the retrieved data easy for information users to view.

[0809] Input: Query results

[0810] What it does: Sorts search results by relevance and formats them in HTML.

[0811] Output: Formatted search results

[0812] The terminal displays the search results, ie, displays the search results returned from the server to the user.

[0813] Input: Formatted search results

[0814] Action: Show search results

[0815] Output: Search results displayed to the user

[0816] In this way, by processing each step sequentially, this system enables information users to efficiently search and retrieve data that is centrally managed across multiple information transmission tools.

[0817] (Application example 1)

[0818] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0819] Previously, maintenance information for machinery and equipment within a factory was managed separately using multiple communication methods, making centralized management of the information difficult. Furthermore, searching and analyzing maintenance information required a great deal of time and effort, making efficient maintenance management difficult. This resulted in delays in responding to machine breakdowns and malfunctions, leading to reduced production efficiency.

[0820] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0821] In this invention, the server includes means for collecting data from multiple communication means, means for storing the collected data in an information storage device, and means for analyzing the stored data using a natural language processing algorithm, thereby enabling the centralized collection and analysis of information related to the maintenance of factory equipment and the prompt provision of necessary information.

[0822] "Multiple means of communication" refers to a variety of digital communication tools such as email, chat, and maintenance logs.

[0823] "Data collection method" refers to the function of periodically obtaining new messages and files using APIs and OAuth tokens of multiple communication methods.

[0824] "Information storage device" refers to a digital storage medium such as a database or cloud storage for storing collected data.

[0825] "Natural language processing algorithm" refers to technology that analyzes text data using methods such as SpaCy and BERT and extracts keywords and contextual information.

[0826] "Automatic classification means" refers to a function that automatically classifies analyzed data based on specific criteria or categories.

[0827] "Attribute information" refers to metadata such as timestamps and sender information that is added to data.

[0828] "Search instructions" refer to keywords or phrases that a user enters into a search screen.

[0829] "Search means" refers to a function that queries an information storage device based on a search instruction entered by a user to retrieve relevant data.

[0830] "Means for displaying in order of relevance" refers to a function for sorting search results in order of relevance and displaying them to the user.

[0831] "Information on the maintenance of factory equipment" refers to data related to the maintenance, inspection, and repair of machinery and equipment within a factory.

[0832] The present invention relates to a system for efficiently managing information relating to the maintenance of factory equipment and for quickly providing necessary information. Specific embodiments of the present invention will be described below.

[0833] First, the server collects data from multiple communication methods required within the factory (e.g., email, chat, maintenance logs). The data is acquired using the API of each communication method and stored in an information storage device (database or cloud storage).

[0834] The server then analyzes the stored data using natural language processing algorithms. Specifically, it uses natural language processing libraries such as SpaCy and BERT to analyze the text data and extract keywords and contextual information. This allows for automatic classification of the data and the addition of attribute information (such as timestamps and sender information).

[0835] Users input search instructions into a search screen using a display or tablet installed on a factory robot. Based on this, the server queries the information storage device and retrieves relevant data. Search results are displayed in order of relevance and presented in a format that is intuitively easy for users to understand.

[0836] For example, if a factory robot searches for "Machine A's maintenance history for the past six months," the system quickly displays detailed historical information, including timestamps and maintenance details. Similarly, if a user searches for "error messages," the system also displays related repair procedures and contact information.

[0837] This system centrally manages information related to factory equipment maintenance and provides necessary information quickly, which speeds up responses to machine breakdowns and malfunctions and improves production efficiency.

[0838] Examples of prompts for a generative AI model might include:

[0839] "We want to develop a system that collects maintenance log data, analyzes its contents, and provides maintenance history and error information. What architecture and technology would be suitable?"

[0840] In this manner, specific elements and procedures are described in detail to clearly provide specific embodiments of the invention.

[0841] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0842] Step 1:

[0843] The server collects data from multiple communication methods within the factory (email, chat, maintenance logs). Specifically, it uses APIs and OAuth tokens to obtain access to each communication method and periodically checks whether there are any new messages or files. The input requires the API key or OAuth token for each communication method, and the output is the collected message and file data.

[0844] Step 2:

[0845] The server stores the collected data in an information storage device (database). Specifically, it extracts attribute information such as the contents of the collected messages and files, timestamps, and sender information, converts it into an appropriate format (e.g., JSON format), and stores it in the database. The input is the data collected in step 1, and the output is the data stored in the information storage device.

[0846] Step 3:

[0847] The server analyzes the data stored in the information storage device using natural language processing algorithms (e.g., SpaCy or BERT). Specifically, it analyzes the text data to extract key keywords and contextual information, and then classifies the data based on that. The input is the unanalyzed data stored in the information storage device, and the output is the analyzed data and its keyword information.

[0848] Step 4:

[0849] The server adds attribute information, such as a timestamp and sender information, to the analyzed data. This allows the analyzed data to be classified in more detail and supplemented with related information. The input is the analyzed data, and the output is the analyzed data with the attribute information added.

[0850] Step 5:

[0851] The user inputs search instructions using a tablet or display installed on the factory robot. Specifically, they input keywords or phrases (e.g., "Maintenance history of machine A") into the search screen. The input is the search instruction entered by the user, and the output is the search instruction itself.

[0852] Step 6:

[0853] The server queries the information storage device based on the search instructions entered by the user. Specifically, it finds data corresponding to the search keywords and retrieves related data. The input is the search instructions entered by the user, and the output is the related data as the search results.

[0854] Step 7:

[0855] The server sorts the retrieved search results in order of relevance and provides them to the user. Specifically, it formats the search results in order of relevance and displays them on the display in a format that is intuitively easy for the user to understand. The input is related data as search results, and the output is a display of the formatted search results.

[0856] Step 8:

[0857] The user selects the required information from the displayed search results and checks the details. Specifically, the user clicks on the relevant information from the list of search results to open the details screen. The input is the list of search results, and the output is the selected detailed information.

[0858] The above steps will realize a system that efficiently collects and manages information related to factory equipment maintenance and quickly provides the necessary information. This system will enable a quick response to machine breakdowns and malfunctions, contributing to improved production efficiency.

[0859] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0860] MODE FOR CARRYING OUT THE INVENTION

[0861] This invention is a system that combines a system that collects data from multiple communication tools, stores the data in a database, and analyzes and classifies it using natural language processing algorithms with an emotion engine that recognizes the user's emotions. Specific embodiments of this system and the program processing procedures are described below.

[0862] 1. Data collection

[0863] The server obtains access using API keys or OAuth tokens from Gmail, GChat, or other communication tools. The server uses these APIs to periodically check for new messages or files. For example, to collect data from Gmail, the server uses the Gmail API to retrieve unread emails and extract the email content, attachments, sent time (timestamp), and sender information. This extracted data is then stored in a database in a data format such as JSON.

[0864] 2. Data Analysis

[0865] The server then analyzes the stored data using natural language processing algorithms (NLP), such as SpaCy or BERT, to extract key keywords and context from the content of message bodies and attachments. This analysis identifies criteria and categories for classifying the data. The server then automatically classifies each message or file based on the extracted keywords and context. Furthermore, the server complements the data with metadata such as timestamps and sender information.

[0866] 3. User Emotion Recognition

[0867] The server uses an emotion engine to recognize the user's emotions when the user accesses the search screen and enters a search term. The emotion engine evaluates the user's emotional state through facial recognition, voice analysis, and text analysis. For example, when a user searches for "Project X report," the emotion engine analyzes the user's voice tone, facial expression, and the tone of the text entered to understand the user's current emotional state.

[0868] 4. Tailor search results based on sentiment

[0869] The server adjusts the order of search results based on the user's emotions, as determined by the emotion engine. For example, if the user is feeling stressed, the server prioritizes relevant results and reduces unnecessary information. If the user is feeling relaxed, the server also displays detailed explanations and related materials.

[0870] 5. Search Results Display and Optimization

[0871] Users can check the search results provided by the server. The server continuously analyzes the user's reactions using an emotion engine and optimizes the display of search results from the next time. For example, if a user has a positive reaction to a particular search result, the server will display results based on that reaction when a similar search is performed.

[0872] Specific examples

[0873] For example, if a user searches for "Project X meeting minutes," the server will search for meeting minutes messages related to Project X and display them in order of relevance along with metadata such as timestamps and sender information. At this time, the emotion engine will adjust the display order of search results as needed based on the user's emotional state. The emotion engine also determines whether the user is satisfied with the search results, which will be used to improve the accuracy of search results in future searches.

[0874] This system allows users to centrally manage what information has been exchanged with which tools, enabling them to efficiently obtain the information they need. In addition, by combining it with an emotion engine, it is possible to display information optimally according to the user's emotional state, further improving work efficiency.

[0875] The processing flow will be explained below.

[0876] Step 1: Collect data

[0877] 1. The server obtains access rights using the API key or OAuth token of each communication tool (e.g., Gmail, GChat).

[0878] 2. The server calls the API of each tool at regular intervals to check whether there are any new messages or files.

[0879] 3. The server extracts the following information from the retrieved message or file:

[0880] Message body

[0881] Attachments

[0882] timestamp (time of sending)

[0883] Sender information (email address, name, etc.)

[0884] 4. The server stores the extracted data in a database in JSON or other data formats.

[0885] Step 2: Analyze the data

[0886] 1. The server analyzes messages and attachments stored in a database using natural language processing algorithms.

[0887] For example, use SpaCy or BERT to extract key keywords and context.

[0888] 2. The server automatically classifies each message or file based on the extracted keywords.

[0889] For example, data related to Project X may be classified based on the keyword "Project X."

[0890] 3. The server adds metadata such as timestamps and sender information to the classified data.

[0891] Step 3: Acquire data for emotion recognition

[0892] 1. The user accesses a dedicated search screen and prepares to enter a search term.

[0893] 2. The device collects the user's facial expression and voice data when the user enters a search term.

[0894] For example, a webcam and microphone are used to capture the user's facial expressions and voice tone.

[0895] 3. The server analyzes the collected emotion data using an emotion engine.

[0896] For example, facial recognition software and voice analysis tools are used to determine a user's emotional state.

[0897] Step 4: Perform sentiment-based search

[0898] 1. The user enters a search term into a dedicated search screen.

[0899] For example, enter "Project X Report."

[0900] 2. The server queries the database using the emotion data from the emotion engine along with the entered search term.

[0901] For example, if you search for "Project X report" and the user is feeling stressed, the system will prioritize the most relevant results.

[0902] Step 5: Formatting and displaying search results

[0903] 1. The server sorts the retrieved search results by relevance.

[0904] For example, the analysis results from the emotion engine are taken into account to display more relevant results at the top.

[0905] 2. The server formats the search results in a format that is easy for the user to understand.

[0906] Each result includes the message body, attachments (with links), timestamps, and sender information.

[0907] 3. The user reviews the organized search results and accesses the information they need.

[0908] Step 6: Collect and optimize responses to search results

[0909] 1. The device collects the user's reactions as they check the search results.

[0910] For example, it monitors the user's facial expressions, voice tone, clicking behavior, etc.

[0911] 2. The server analyzes the reaction data with an emotion engine to evaluate how users reacted to a particular search result.

[0912] 3. The server will optimize the search results displayed from the next time onwards based on the analysis results.

[0913] For example, the search algorithm can be adjusted to prioritize results that satisfy users.

[0914] This allows the user to efficiently obtain the necessary information and enables optimal information display according to the user's emotional state.

[0915] Example 2

[0916] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0917] In today's business environment, a large amount of information is exchanged through multiple communication tools, creating a need for centralized data management and efficient information search. However, due to a lack of optimization of search results that take into account the user's emotional state, it is difficult to provide appropriate information according to the user's state. This leads to problems such as reduced work efficiency and increased stress for users.

[0918] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0919] In this invention, the server includes means for collecting data from multiple communication tools, means for saving the collected data in a database, means for analyzing the saved data using a natural language processing algorithm, means for automatically classifying the analyzed data, means for adding metadata to the classified data, means for searching the database based on search words entered by a user on a search screen, means for displaying search results in order of relevance, means for using an emotion engine that recognizes the user's emotions when the user accesses the search screen, means for adjusting the display order of search results based on the emotion information recognized by the emotion engine, and means for continuously analyzing the user's reactions using the emotion engine and optimizing the display of search results from the next time onwards. This enables centralized management of information and provision of appropriate information according to the user's emotional state.

[0920] "Multiple communication tools" refers to various means by which users exchange messages and data electronically, such as email and chat applications.

[0921] "Means for collecting data" refers to software and hardware components used to obtain message and file information from designated communication tools.

[0922] "Means for storing data in a database" refers to a storage device or management system for permanently storing collected data.

[0923] "Natural language processing algorithms" refer to machine learning and language models that analyze collected text data and extract context and meaning.

[0924] "Means for automatically classifying data" refers to a function that divides data into specific categories or tags based on the results of analysis.

[0925] "Metadata" refers to additional information about a message or file (e.g., timestamps and originator information).

[0926] A "search screen" refers to an interface that allows a user to input search keywords and search for information in a database.

[0927] "Emotion engine" refers to technology that performs facial recognition, voice analysis, and text analysis of a user to assess their emotional state.

[0928] "Means for adjusting the display order of search results" refers to a function for sorting search results in an appropriate order based on the user's emotional state.

[0929] "Means for continuously analyzing user responses" refers to technology that collects user search activity and responses to the results, and optimizes the display of search results from the next time onwards.

[0930] MODE FOR CARRYING OUT THE INVENTION

[0931] This invention is a system that combines a system that collects data from multiple communication tools, stores the data in a database, and analyzes and classifies the data using a natural language processing algorithm with an emotion engine that recognizes the user's emotions. Specific embodiments for implementing this system will be described below.

[0932] 1. Data collection

[0933] The server obtains access rights from multiple communication tools, such as Gmail and chat tools, using API keys and OAuth tokens. It periodically checks for new messages and files using the Gmail API and chat API. For example, when collecting data from Gmail, the server retrieves unread emails and extracts the email content, attachments, sent date and time, and sender information. This extracted data is stored in a database in a data format such as JSON.

[0934] 2. Data Analysis

[0935] The server then analyzes the stored data using natural language processing algorithms, particularly tools like SpaCy and BERT. These tools extract key keywords and context from the content of messages and attachments. This analysis determines the criteria and categories for categorizing the data. The server automatically classifies each message or file based on the extracted keywords and context. It also adds metadata such as timestamps and sender information to complement the data.

[0936] 3. User Emotion Recognition

[0937] When a user accesses the search screen from their device and enters a search term, the server uses an emotion engine to recognize the user's emotion. The emotion engine evaluates the user's emotional state through facial recognition, voice analysis, and text analysis. For example, when a user searches for "Project X Report," the server analyzes the user's voice tone, facial expression, and the tone of the text entered to understand the user's current emotional state.

[0938] 4. Tailor search results based on sentiment

[0939] The server adjusts the display order of search results based on the emotional information recognized by the emotion engine. For example, if the user is feeling stressed, the server will prioritize relevant results and reduce unnecessary information. If the user is feeling relaxed, the server will also display detailed explanations and related materials.

[0940] 5. Search Results Display and Optimization

[0941] The server displays the adjusted search results on the user's device. The user can then review the displayed search results. Furthermore, the server continuously analyzes the user's reactions using an emotion engine to optimize the display of search results from the next time onwards. For example, if a user has a positive reaction to a particular search result, the server will display results based on that reaction when a similar search is performed.

[0942] Examples of concrete examples and prompts

[0943] For example, consider the case where a user searches for "Project X meeting minutes." The server searches the database for messages of meeting minutes related to Project X and sorts them in order of relevance along with metadata such as timestamps and sender information. The emotion engine then adjusts the display order of the search results as needed based on the user's emotional state. The emotion engine also analyzes the user's reaction to the search results and optimizes the display of search results from the next time onwards.

[0944] An example of a prompt sentence might be "Please search for Project X meeting minutes."

[0945] This system allows users to centrally manage what information has been exchanged using which tools, enabling them to efficiently obtain the information they need. In addition, the introduction of an emotion engine makes it possible to provide optimal information according to the user's emotional state, significantly improving work efficiency.

[0946] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0947] Step 1: Collect data

[0948] 1.1 The server obtains API keys and OAuth tokens from multiple communication tools such as Gmail and chat tools. The input is the authentication information of each tool, and the output is the obtained access rights information.

[0949] 1.2 The server uses the obtained access right information to access each tool and periodically checks whether new messages or files exist. The input is the access right information, and the output is the new message or file information.

[0950] 1.3 The server retrieves unread emails using the Gmail API and extracts the email content, attachments, sent date and time, and sender information. The input is access rights information and new message information, and the output is the extracted email data.

[0951] 1.4 The server saves the extracted data in a database in JSON format. The input is the extracted email data, and the output is the data saved in the database.

[0952] Step 2: Analyze the data

[0953] 2.1 The server analyzes the stored data based on a natural language processing algorithm, where the input is the stored data and the output is the analysis result.

[0954] 2.2 The server uses natural language processing tools such as SpaCy or BERT to extract key keywords and context from the content of the message body and attachments. The input is the stored data, and the output is the extracted keywords and context.

[0955] 2.3 The server automatically classifies each message or file based on the extracted keywords and context. The input is the extracted keywords and context, and the output is the classified data.

[0956] 2.4 The server complements the data by adding metadata such as timestamps and sender information. The input is the classified data, and the output is the data with the metadata added.

[0957] Step 3: Recognizing user emotions

[0958] 3.1 The user accesses the search screen from their device and enters a search term. The input is the search term, and the output is the transmission of the search term.

[0959] 3.2 The server passes the user's input to the emotion engine. The input is the search word, and the output is the input to the emotion engine.

[0960] 3.3 The emotion engine evaluates the user's emotional state through facial recognition, voice analysis, and text analysis. The input is the search term and the user's biometric information, and the output is the user's emotional state.

[0961] 3.4 For example, when a user searches for "Project X report," the emotion engine analyzes the user's tone of voice, facial expression, and the tone of the text entered to understand the user's current emotional state. The input is the search word and the user's biometric information, and the output is the analyzed emotional state.

[0962] Step 4: Tailor search results based on sentiment

[0963] 4.1 The server adjusts the display order of search results based on the user's emotional information recognized by the emotion engine. The input is the user's emotional information, and the output is the adjusted search results.

[0964] 4.2 For example, if a user is feeling stressed, the server prioritizes relevant results and reduces unnecessary information. The input is the user's emotional information, and the output is the prioritized search results.

[0965] 4.3 If the user is relaxed, detailed explanations and related materials are also displayed. The input is the user's emotional information, and the output is search results with added detailed information.

[0966] Step 5: Displaying and optimizing search results

[0967] 5.1 The server displays the tailored search results on the user's terminal. The input is the tailored search results and the output is the results displayed on the terminal.

[0968] 5.2 The user can confirm the search results displayed on the terminal. The input is the displayed search results, and the output is the user's confirmation.

[0969] 5.3 The server analyzes user responses to search results (clicks, time spent, etc.) using an emotion engine. The input is the user response, and the output is the analysis result.

[0970] 5.4 The server will optimize the display of search results from the next time onwards based on the analysis results. The input is the analysis results, and the output is the optimized search result display format.

[0971] This series of processing steps allows the system to provide optimal information according to the user's emotional state, allowing the user to efficiently obtain the information they need and improve their work efficiency.

[0972] (Application example 2)

[0973] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0974] In recent years, brick-and-mortar stores have been required to provide services that respond promptly to customer needs, but it is difficult to grasp the emotional state of customers in real time, making it difficult to provide optimal customer service and product recommendations. Furthermore, there is a demand for methods to improve customer satisfaction by utilizing past communication data, but there is a lack of efficient ways to do this. There is an urgent need to provide a system that improves customer experience by resolving these issues.

[0975] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0976] In this invention, the server includes means for collecting data from multiple communication tools, means for saving the collected data in a database, means for analyzing the saved data using a natural language processing algorithm, means for automatically classifying the analyzed data, means for adding metadata to the classified data, means for searching the database based on search words entered by a user on a search screen, means for displaying search results in order of relevance, means for recognizing the user's emotions in real time, and means for adjusting the display order of information based on the recognized emotions. This makes it possible to grasp the emotional state of customers in real time and provide optimal customer service and product suggestions based on past communication data.

[0977] 1. "Communication tools" is a general term for software and hardware used to send and receive information via digital media, such as email and chat applications.

[0978] 2. "Database" means a data management system that efficiently manages collected data and allows for searching and analysis as needed.

[0979] 3. "Natural language processing algorithm" is a general term for algorithms that analyze text data and understand and process its meaning.

[0980] 4. "Classification" is the process of grouping analyzed data based on specific criteria.

[0981] 5. "Metadata" is a general term for attribute information and additional information related to data, which is added to facilitate data search and management.

[0982] 6. "Search screen" is an interface that allows users to enter keywords and search for information in the database.

[0983] 7. "Relevance" is a criterion for sorting search results based on the degree of match or importance with the keywords entered by the user.

[0984] 8. "Emotion recognition means" is a general term for technologies and devices that analyze a user's facial expressions and vocal tone to understand their emotional state.

[0985] 9. "Adjusting the information display order" is the process of changing the order of displayed information based on perceived user sentiment.

[0986] This invention is a system for grasping the emotional state of customers in real time in a physical store and providing optimal customer service and product recommendations. This system collects data from multiple communication tools and analyzes it with a natural language processing algorithm to grasp the user's emotional state in real time and adjust the order in which information is displayed based on this data.

[0987] First, the server collects data from communication tools such as Gmail and chat apps. The server uses the API keys or OAuth tokens of these tools to gain access and periodically checks for new messages and files. The collected data is stored in a database in a data format such as JSON, allowing salespeople to quickly reference past interactions with customers.

[0988] The server then analyzes the stored data using natural language processing algorithms. Generative AI models like SpaCy and BERT are used to extract key keywords and context from the content of messages and attachments. This analysis identifies criteria or categories for classifying the data. Classification tags and metadata are then added to the analyzed data.

[0989] Additionally, the system employs emotion recognition to recognize users' emotions in real time. This is achieved by using the camera and microphone built into the smart glasses to analyze the customer's facial expressions and tone of voice. For emotion recognition, the system uses the Hugging Face Transformers library to evaluate their emotional state. This emotion data is stored in the cloud and can be used to trigger specific actions.

[0990] The system also provides a means to adjust the order in which information is displayed based on emotions. For example, if a customer is feeling stressed, relevant search results will be displayed first, reducing unnecessary information. On the other hand, if the customer is relaxed, detailed explanations and related materials will also be displayed. This adjustment makes it possible to provide the optimal service according to the customer's state.

[0991] The following specific examples are given as examples.

[0992] Example 1: When a customer searches for "Project X meeting minutes"

[0993] The server searches for meeting minutes related to Project X and displays them in order of relevance along with timestamps and sender information. The emotion engine adjusts the display order according to the user's emotional state and provides detailed information as needed.

[0994] Example 2: When a customer is stressed

[0995] Prompt: "What is the appropriate way to respond when a customer is stressed?"

[0996] Example output: "Respond simply and quickly, and refrain from suggesting new products."

[0997] Example 3: When the customer is relaxed

[0998] Prompt: "What is the appropriate way to respond when a customer is relaxed?"

[0999] Sample output: "Provides detailed product descriptions and offers additional options."

[1000] This will enable customer service in physical stores to respond more quickly to the emotional state of customers and provide greater satisfaction.

[1001] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1002] Step 1:

[1003] The server obtains access rights using API keys or OAuth tokens from communication tools such as Gmail and chat apps. The input is the authentication information for each communication tool, and the output is the access rights to these tools. Specifically, the server periodically checks for and collects unread emails using the Gmail API.

[1004] Step 2:

[1005] The server collects new messages and files from the communication tools it has gained access to and stores them in a database in an appropriate format, such as JSON. The input is the data of the new messages and files, and the output is the data stored in the database. Specifically, timestamps and sender information are also stored in the database at the same time.

[1006] Step 3:

[1007] The server analyzes the stored data using natural language processing algorithms (e.g., SpaCy or BERT). The input is the text data stored in the database, and the output is the analysis results, including key keywords and contextual information. Specifically, meaning is extracted from the contents of the message body and attachments and automatically classified.

[1008] Step 4:

[1009] The server adds classification tags and metadata to the analyzed data. The input is the main keywords and context information from the analysis results, and the output is data with the classification tags and metadata added. Specifically, category information, timestamp, and sender information are added to each data.

[1010] Step 5:

[1011] A user enters keywords into a search screen, and the server searches the database based on those keywords. The input is the search keyword entered by the user, and the output is a list of related data. Specifically, the server extracts highly relevant data from the searched data.

[1012] Step 6:

[1013] The server sorts the search results in order of relevance and displays them to the user. The input is a list of searched data, and the output is the search results sorted in order of relevance. Specifically, the list is displayed taking into account related metadata.

[1014] Step 7:

[1015] The system grasps the user's emotional state in real time through a camera and microphone connected to a device (such as smart glasses). The input is camera video and microphone audio, and the output is the analysis result of the user's emotional state. Specifically, it uses the Hugging Face Transformers library to recognize emotions from facial expressions and voice tone.

[1016] Step 8:

[1017] The server adjusts the display order of information based on the recognized emotional state. The input is the analysis result of the user's emotional state and the search results, and the output is the adjusted display order of information. Specifically, if the user is feeling stressed, highly relevant information is displayed first, and if the user is relaxed, additional detailed information is displayed.

[1018] Step 9:

[1019] The server continuously analyzes customer responses using an emotion engine to optimize the display of search results from the next time onwards. The input is customer response data, and the output is search results in an optimized display order. Specifically, the next display is improved based on search results that have generated positive responses.

[1020] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1021] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1022] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1023] [Fourth embodiment]

[1024] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1025] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1027] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1028] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1029] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1031] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1032] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1033] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1035] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1037] MODE FOR CARRYING OUT THE INVENTION

[1038] This invention is a system that collects data from multiple communication tools, stores it in a database, analyzes the data using a natural language processing algorithm, and provides related information based on search terms. Specific embodiments of this system and the program processing procedures are described below.

[1039] 1. Data collection

[1040] First, the server gets access to Gmail, GChat, and other communication tools using their API keys or OAuth tokens, and then periodically checks for new messages and files.

[1041] For example, if a server collects data from Gmail, it uses the Gmail API to retrieve unread emails and extracts the email content, attachments, sent time (timestamp), and sender information. This extracted data is then stored in a database in a data format such as JSON.

[1042] 2. Data Analysis

[1043] The server then analyzes the stored data using natural language processing algorithms (NLP), such as SpaCy or BERT, to extract key keywords and context from the content of the message body and attachments. This analysis identifies criteria and categories for classifying the data.

[1044] The server automatically categorizes each message or file based on extracted keywords and context, and complements the data with metadata such as timestamps and sender information.

[1045] 3. Search and display in the user interface

[1046] Users use a dedicated search screen to enter search terms to find the information they need. The server queries the database based on the search terms and retrieves related messages and files. The retrieved search results are sorted by relevance and displayed in an easy-to-read format for the user.

[1047] For example, if a user searches for "Project X report," the server will search for report messages related to Project X and display them in order of relevance, along with metadata such as timestamps and sender information. The search results include the message body, links to attachments, time of sending, and sender information.

[1048] Specific examples

[1049] As a concrete example, let's consider the case where a server collects data from G Chat. In this case, the server first obtains new messages using the G Chat API. The server then extracts the contents, attachments, timestamps, sender information, etc. from the obtained messages and stores them in a database.

[1050] The server then analyzes this data and categorizes it into categories, such as "progress reports" or "meeting minutes." If a user searches for "meeting minutes," messages and files related to that search term are displayed. The server displays results based on chronology and importance, allowing users to quickly access the information they need.

[1051] This system allows users to centrally manage what information has been exchanged with which tools, and efficiently obtain the information they need, greatly improving work efficiency.

[1052] The processing flow will be explained below.

[1053] Step 1: Collect data

[1054] 1. The server obtains access rights using the API key or OAuth token of each communication tool (e.g., Gmail, GChat).

[1055] 2. The server calls the API of each tool at regular intervals to check whether there are any new messages or files.

[1056] 3. The server extracts the following information from the retrieved message or file:

[1057] Message body

[1058] Attachments

[1059] timestamp (time of sending)

[1060] Sender information (email address, name, etc.)

[1061] 4. The server stores the extracted data in a database in JSON or other data formats.

[1062] Step 2: Analyze the data

[1063] 1. The server analyzes messages and attachments stored in a database using natural language processing algorithms.

[1064] Example: Use SpaCy or BERT to extract key keywords and context.

[1065] 2. The server automatically classifies each message or file based on the extracted keywords.

[1066] Example: Classify data related to Project X based on the keyword "Project X".

[1067] 3. The server adds metadata such as timestamps and sender information to the classified data.

[1068] Step 3: Perform a search

[1069] 1. The user accesses a dedicated search screen and enters a search term.

[1070] For example, enter "Project X Report."

[1071] 2. The server queries the database based on the entered search term.

[1072] Example: Search for "Project X Report" to get relevant data.

[1073] Step 4: Formatting and displaying search results

[1074] 1. The server sorts the retrieved search results by relevance.

[1075] Example: sorting search results based on relevance score.

[1076] 2. The server formats the search results in a format that is easy for the user to understand.

[1077] Each result includes the message body, attachments (with links), timestamps, and sender information.

[1078] 3. The user reviews the organized search results and accesses the information they need.

[1079] Through these steps, users can centrally manage information that is scattered across multiple communication tools and efficiently obtain the information they need.

[1080] Example 1

[1081] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1082] In today's world, it is common for people to use multiple information transmission tools for business and communication. However, it is difficult to centrally manage and analyze data from these information transmission tools and efficiently search and acquire necessary information. In particular, there is a demand for a system that can uniformly process data in different formats and quickly extract highly relevant information. The objective of this invention is to provide a system that solves these data management and analysis problems and enables users to efficiently acquire necessary information.

[1083] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1084] In this invention, the server includes means for collecting data from multiple information transmission tools, means for storing the collected data in a data management device, means for analyzing the stored data using a natural language processing algorithm, means for automatically classifying the analyzed data, means for adding metadata to the classified data, means for searching the data management device based on search terms entered by an information user on a search screen, means for displaying search results in order of relevance, means for using SpaCy or BERT as a natural language processing algorithm, means for storing the collected data in JSON format, and means for periodically checking for new messages and files from the information transmission tools. This enables information users to efficiently search and retrieve data centrally managed by different information transmission tools.

[1085] An "information communication tool" is a system for sending and receiving messages or files in digital form, such as email, chat applications, or online messaging systems.

[1086] A "data management device" is a database or storage system for storing, managing, and searching collected data.

[1087] "Natural language processing algorithms" are algorithms used to analyze text data, extract keywords, understand context, classify categories, etc. Examples include SpaCy and BERT.

[1088] "Metadata" refers to attribute information of data, and includes, for example, the date and time of data creation, sender information, related categories, and the like.

[1089] The "search screen" is a user interface that allows an information user to input a search word and execute a search query on the data management device.

[1090] "Relevance order" refers to the order in which search results are sorted in order of the most relevant to the information user's search word.

[1091] "SpaCy" is an open source Python library for implementing advanced natural language processing.

[1092] "BERT" is a transformer-based natural language processing model and an algorithm for performing highly accurate text analysis.

[1093] "JSON format" is an abbreviation for JavaScript Object Notation, and is a data format that represents structured data in a text format that is easy for humans to read and machine to process.

[1094] "Periodic check for new messages and files" refers to the process where the server calls the API at regular intervals to check for newly added messages and files.

[1095] MODE FOR CARRYING OUT THE INVENTION

[1096] The present invention provides a system that collects data from multiple information transmission tools, stores it in a data management device, analyzes the data using a natural language processing algorithm, and provides related information based on search terms. This system allows information users to efficiently and quickly obtain the information they need.

[1097] First, the server collects data from multiple communication tools, including email, chat applications, and online messaging systems. In this process, the server obtains API access using each tool's API key or OAuth token. For example, it periodically checks for new messages and files using the APIs of Gmail and chat applications to obtain data. The obtained data is then stored in a data management device in JSON format.

[1098] The server then analyzes the stored data using natural language processing algorithms. This analysis uses advanced natural language processing libraries such as SpaCy and BERT. The server tokenizes the text data and automatically classifies it by extracting keywords and performing contextual analysis. Key keywords and context are extracted from the content of messages and files, and they are then classified into categories such as "progress report" or "meeting minutes." Metadata such as timestamps and sender information is also added to the data.

[1099] The information user then uses a dedicated search screen to enter search terms to find the information they need. The device sends the search terms to the server, which then queries the data management device to retrieve related messages and files. The retrieved search results are sorted by relevance and displayed in a format that is easy for the information user to view. For example, if the information user searches for "Project X report," the server will display a list of reports and emails related to Project X, providing the message body, links to attachments, the time of sending, sender information, and more.

[1100] As a concrete example, let us consider the case where a server collects data from a chat application. The server first uses the chat application's API to obtain new messages. It extracts the content, attachments, timestamps, and sender information from the obtained messages and stores them in a data management device. The server then analyzes this data and classifies it into categories such as "progress reports" and "meeting minutes." When an information user searches for "meeting minutes," messages and files related to that search term are displayed. The server displays the search results in chronological order and by importance, allowing the information user to quickly access the information they need.

[1101] Examples of prompts for generative AI models include:

[1102] How can I get unread emails using the Email API?

[1103] "Explain how to extract keywords from text using a natural language processing library."

[1104] "What's the best way to construct a search query?"

[1105] In this way, by using this system, information users can efficiently search and retrieve data that is centrally managed across multiple information transmission tools, significantly improving work efficiency.

[1106] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1107] Step 1: Collect data

[1108] The server establishes API access. Specifically, it obtains API access rights using the API key or OAuth token of an information transmission tool such as Gmail or a chat application.

[1109] Input: API key or OAuth token of the information transmission tool

[1110] How it works: Obtain an access token through the API authentication process

[1111] Output: Access token

[1112] The server acquires data by collecting new messages and files from each information transmission tool via API.

[1113] Input: Access token, API endpoint

[1114] What it does: For example, call an endpoint that uses the Gmail API to get unread emails.

[1115] Output: Collected data (message body, attachments, timestamp, sender information)

[1116] The server saves the collected data in a data management device. The collected data is stored in the data management device in JSON format.

[1117] Input: Collected data (JSON format)

[1118] Action: Executes an INSERT statement to the database and saves the data.

[1119] Output: Data stored in the data management device

[1120] Step 2: Analyze the data

[1121] The server applies natural language processing algorithms, using libraries such as SpaCy and BERT, to analyze the stored data.

[1122] Input: Text data stored in the data management device

[1123] Operation: Tokenizes text data, extracts keywords, and performs contextual analysis

[1124] Output: Analysis results (extracted keywords and context information)

[1125] The server then categorizes the data based on the analysis results, categorizing messages and files based on extracted keywords and context.

[1126] Input: Analysis results

[1127] Behavior: For example, classifying data into categories such as "progress reports" and "meeting minutes"

[1128] Output: Classified data

[1129] The server adds metadata, such as a timestamp and sender information, to the data.

[1130] Input: Classified data, timestamp information, caller information

[1131] Behavior: Adds metadata to each data entry

[1132] Output: Data with metadata

[1133] Step 3: Search and display in the user interface

[1134] Users use a dedicated search screen to input search terms to find the information they need.

[1135] Input: Search word

[1136] Operation: The device sends the search word to the server.

[1137] Output: Search request to the server

[1138] The server performs a database query, querying the data management device based on the entered search words.

[1139] Input: Search word

[1140] Operation: Searches for the relevant data using the SQL SELECT statement

[1141] Output: Query results (associated messages and files)

[1142] The server formats the search results in order of relevance, making the retrieved data easy for information users to view.

[1143] Input: Query results

[1144] What it does: Sorts search results by relevance and formats them in HTML.

[1145] Output: Formatted search results

[1146] The terminal displays the search results, ie, displays the search results returned from the server to the user.

[1147] Input: Formatted search results

[1148] Action: Show search results

[1149] Output: Search results displayed to the user

[1150] In this way, by processing each step sequentially, this system enables information users to efficiently search and retrieve data that is centrally managed across multiple information transmission tools.

[1151] (Application example 1)

[1152] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1153] Previously, maintenance information for machinery and equipment within a factory was managed separately using multiple communication methods, making centralized management of the information difficult. Furthermore, searching and analyzing maintenance information required a great deal of time and effort, making efficient maintenance management difficult. This resulted in delays in responding to machine breakdowns and malfunctions, leading to reduced production efficiency.

[1154] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1155] In this invention, the server includes means for collecting data from multiple communication means, means for storing the collected data in an information storage device, and means for analyzing the stored data using a natural language processing algorithm, thereby enabling the centralized collection and analysis of information related to the maintenance of factory equipment and the prompt provision of necessary information.

[1156] "Multiple means of communication" refers to a variety of digital communication tools such as email, chat, and maintenance logs.

[1157] "Data collection method" refers to the function of periodically obtaining new messages and files using APIs and OAuth tokens of multiple communication methods.

[1158] "Information storage device" refers to a digital storage medium such as a database or cloud storage for storing collected data.

[1159] "Natural language processing algorithm" refers to technology that analyzes text data using methods such as SpaCy and BERT and extracts keywords and contextual information.

[1160] "Automatic classification means" refers to a function that automatically classifies analyzed data based on specific criteria or categories.

[1161] "Attribute information" refers to metadata such as timestamps and sender information that is added to data.

[1162] "Search instructions" refer to keywords or phrases that a user enters into a search screen.

[1163] "Search means" refers to a function that queries an information storage device based on a search instruction entered by a user to retrieve relevant data.

[1164] "Means for displaying in order of relevance" refers to a function for sorting search results in order of relevance and displaying them to the user.

[1165] "Information on the maintenance of factory equipment" refers to data related to the maintenance, inspection, and repair of machinery and equipment within a factory.

[1166] The present invention relates to a system for efficiently managing information relating to the maintenance of factory equipment and for quickly providing necessary information. Specific embodiments of the present invention will be described below.

[1167] First, the server collects data from multiple communication methods required within the factory (e.g., email, chat, maintenance logs). The data is acquired using the API of each communication method and stored in an information storage device (database or cloud storage).

[1168] The server then analyzes the stored data using natural language processing algorithms. Specifically, it uses natural language processing libraries such as SpaCy and BERT to analyze the text data and extract keywords and contextual information. This allows for automatic classification of the data and the addition of attribute information (such as timestamps and sender information).

[1169] Users input search instructions into a search screen using a display or tablet installed on a factory robot. Based on this, the server queries the information storage device and retrieves relevant data. Search results are displayed in order of relevance and presented in a format that is intuitively easy for users to understand.

[1170] For example, if a factory robot searches for "Machine A's maintenance history for the past six months," the system quickly displays detailed historical information, including timestamps and maintenance details. Similarly, if a user searches for "error messages," the system also displays related repair procedures and contact information.

[1171] This system centrally manages information related to factory equipment maintenance and provides necessary information quickly, which speeds up responses to machine breakdowns and malfunctions and improves production efficiency.

[1172] Examples of prompts for a generative AI model might include:

[1173] "We want to develop a system that collects maintenance log data, analyzes its contents, and provides maintenance history and error information. What architecture and technology would be suitable?"

[1174] In this manner, specific elements and procedures are described in detail to clearly provide specific embodiments of the invention.

[1175] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1176] Step 1:

[1177] The server collects data from multiple communication methods within the factory (email, chat, maintenance logs). Specifically, it uses APIs and OAuth tokens to obtain access to each communication method and periodically checks whether there are any new messages or files. The input requires the API key or OAuth token for each communication method, and the output is the collected message and file data.

[1178] Step 2:

[1179] The server stores the collected data in an information storage device (database). Specifically, it extracts attribute information such as the contents of the collected messages and files, timestamps, and sender information, converts it into an appropriate format (e.g., JSON format), and stores it in the database. The input is the data collected in step 1, and the output is the data stored in the information storage device.

[1180] Step 3:

[1181] The server analyzes the data stored in the information storage device using natural language processing algorithms (e.g., SpaCy or BERT). Specifically, it analyzes the text data to extract key keywords and contextual information, and then classifies the data based on that. The input is the unanalyzed data stored in the information storage device, and the output is the analyzed data and its keyword information.

[1182] Step 4:

[1183] The server adds attribute information, such as a timestamp and sender information, to the analyzed data. This allows the analyzed data to be classified in more detail and supplemented with related information. The input is the analyzed data, and the output is the analyzed data with the attribute information added.

[1184] Step 5:

[1185] The user inputs search instructions using a tablet or display installed on the factory robot. Specifically, they input keywords or phrases (e.g., "Maintenance history of machine A") into the search screen. The input is the search instruction entered by the user, and the output is the search instruction itself.

[1186] Step 6:

[1187] The server queries the information storage device based on the search instructions entered by the user. Specifically, it finds data corresponding to the search keywords and retrieves related data. The input is the search instructions entered by the user, and the output is the related data as the search results.

[1188] Step 7:

[1189] The server sorts the retrieved search results in order of relevance and provides them to the user. Specifically, it formats the search results in order of relevance and displays them on the display in a format that is intuitively easy for the user to understand. The input is related data as search results, and the output is a display of the formatted search results.

[1190] Step 8:

[1191] The user selects the required information from the displayed search results and checks the details. Specifically, the user clicks on the relevant information from the list of search results to open the details screen. The input is the list of search results, and the output is the selected detailed information.

[1192] The above steps will realize a system that efficiently collects and manages information related to factory equipment maintenance and quickly provides the necessary information. This system will enable a quick response to machine breakdowns and malfunctions, contributing to improved production efficiency.

[1193] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1194] MODE FOR CARRYING OUT THE INVENTION

[1195] This invention is a system that combines a system that collects data from multiple communication tools, stores the data in a database, and analyzes and classifies it using natural language processing algorithms with an emotion engine that recognizes the user's emotions. Specific embodiments of this system and the program processing procedures are described below.

[1196] 1. Data collection

[1197] The server obtains access using API keys or OAuth tokens from Gmail, GChat, or other communication tools. The server uses these APIs to periodically check for new messages or files. For example, to collect data from Gmail, the server uses the Gmail API to retrieve unread emails and extract the email content, attachments, sent time (timestamp), and sender information. This extracted data is then stored in a database in a data format such as JSON.

[1198] 2. Data Analysis

[1199] The server then analyzes the stored data using natural language processing algorithms (NLP), such as SpaCy or BERT, to extract key keywords and context from the content of message bodies and attachments. This analysis identifies criteria and categories for classifying the data. The server then automatically classifies each message or file based on the extracted keywords and context. Furthermore, the server complements the data with metadata such as timestamps and sender information.

[1200] 3. User Emotion Recognition

[1201] The server uses an emotion engine to recognize the user's emotions when the user accesses the search screen and enters a search term. The emotion engine evaluates the user's emotional state through facial recognition, voice analysis, and text analysis. For example, when a user searches for "Project X report," the emotion engine analyzes the user's voice tone, facial expression, and the tone of the text entered to understand the user's current emotional state.

[1202] 4. Tailor search results based on sentiment

[1203] The server adjusts the order of search results based on the user's emotions, as determined by the emotion engine. For example, if the user is feeling stressed, the server prioritizes relevant results and reduces unnecessary information. If the user is feeling relaxed, the server also displays detailed explanations and related materials.

[1204] 5. Search Results Display and Optimization

[1205] Users can check the search results provided by the server. The server continuously analyzes the user's reactions using an emotion engine and optimizes the display of search results from the next time. For example, if a user has a positive reaction to a particular search result, the server will display results based on that reaction when a similar search is performed.

[1206] Specific examples

[1207] For example, if a user searches for "Project X meeting minutes," the server will search for meeting minutes messages related to Project X and display them in order of relevance along with metadata such as timestamps and sender information. At this time, the emotion engine will adjust the display order of search results as needed based on the user's emotional state. The emotion engine also determines whether the user is satisfied with the search results, which will be used to improve the accuracy of search results in future searches.

[1208] This system allows users to centrally manage what information has been exchanged with which tools, enabling them to efficiently obtain the information they need. In addition, by combining it with an emotion engine, it is possible to display information optimally according to the user's emotional state, further improving work efficiency.

[1209] The processing flow will be explained below.

[1210] Step 1: Collect data

[1211] 1. The server obtains access rights using the API key or OAuth token of each communication tool (e.g., Gmail, GChat).

[1212] 2. The server calls the API of each tool at regular intervals to check whether there are any new messages or files.

[1213] 3. The server extracts the following information from the retrieved message or file:

[1214] Message body

[1215] Attachments

[1216] timestamp (time of sending)

[1217] Sender information (email address, name, etc.)

[1218] 4. The server stores the extracted data in a database in JSON or other data formats.

[1219] Step 2: Analyze the data

[1220] 1. The server analyzes messages and attachments stored in a database using natural language processing algorithms.

[1221] For example, use SpaCy or BERT to extract key keywords and context.

[1222] 2. The server automatically classifies each message or file based on the extracted keywords.

[1223] For example, data related to Project X may be classified based on the keyword "Project X."

[1224] 3. The server adds metadata such as timestamps and sender information to the classified data.

[1225] Step 3: Acquire data for emotion recognition

[1226] 1. The user accesses a dedicated search screen and prepares to enter a search term.

[1227] 2. The device collects the user's facial expression and voice data when the user enters a search term.

[1228] For example, a webcam and microphone are used to capture the user's facial expressions and voice tone.

[1229] 3. The server analyzes the collected emotion data using an emotion engine.

[1230] For example, facial recognition software and voice analysis tools are used to determine a user's emotional state.

[1231] Step 4: Perform sentiment-based search

[1232] 1. The user enters a search term into a dedicated search screen.

[1233] For example, enter "Project X Report."

[1234] 2. The server queries the database using the emotion data from the emotion engine along with the entered search term.

[1235] For example, if you search for "Project X report" and the user is feeling stressed, the system will prioritize the most relevant results.

[1236] Step 5: Formatting and displaying search results

[1237] 1. The server sorts the retrieved search results by relevance.

[1238] For example, the analysis results from the emotion engine are taken into account to display more relevant results at the top.

[1239] 2. The server formats the search results in a format that is easy for the user to understand.

[1240] Each result includes the message body, attachments (with links), timestamps, and sender information.

[1241] 3. The user reviews the organized search results and accesses the information they need.

[1242] Step 6: Collect and optimize responses to search results

[1243] 1. The device collects the user's reactions as they check the search results.

[1244] For example, it monitors the user's facial expressions, voice tone, clicking behavior, etc.

[1245] 2. The server analyzes the reaction data with an emotion engine to evaluate how users reacted to a particular search result.

[1246] 3. The server will optimize the search results displayed from the next time onwards based on the analysis results.

[1247] For example, the search algorithm can be adjusted to prioritize results that satisfy users.

[1248] This allows the user to efficiently obtain the necessary information and enables optimal information display according to the user's emotional state.

[1249] Example 2

[1250] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1251] In today's business environment, a large amount of information is exchanged through multiple communication tools, creating a need for centralized data management and efficient information search. However, due to a lack of optimization of search results that take into account the user's emotional state, it is difficult to provide appropriate information according to the user's state. This leads to problems such as reduced work efficiency and increased stress for users.

[1252] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1253] In this invention, the server includes means for collecting data from multiple communication tools, means for saving the collected data in a database, means for analyzing the saved data using a natural language processing algorithm, means for automatically classifying the analyzed data, means for adding metadata to the classified data, means for searching the database based on search words entered by a user on a search screen, means for displaying search results in order of relevance, means for using an emotion engine that recognizes the user's emotions when the user accesses the search screen, means for adjusting the display order of search results based on the emotion information recognized by the emotion engine, and means for continuously analyzing the user's reactions using the emotion engine and optimizing the display of search results from the next time onwards. This enables centralized management of information and provision of appropriate information according to the user's emotional state.

[1254] "Multiple communication tools" refers to various means by which users exchange messages and data electronically, such as email and chat applications.

[1255] "Means for collecting data" refers to software and hardware components used to obtain message and file information from designated communication tools.

[1256] "Means for storing data in a database" refers to a storage device or management system for permanently storing collected data.

[1257] "Natural language processing algorithms" refer to machine learning and language models that analyze collected text data and extract context and meaning.

[1258] "Means for automatically classifying data" refers to a function that divides data into specific categories or tags based on the results of analysis.

[1259] "Metadata" refers to additional information about a message or file (e.g., timestamps and originator information).

[1260] A "search screen" refers to an interface that allows a user to input search keywords and search for information in a database.

[1261] "Emotion engine" refers to technology that performs facial recognition, voice analysis, and text analysis of a user to assess their emotional state.

[1262] "Means for adjusting the display order of search results" refers to a function for sorting search results in an appropriate order based on the user's emotional state.

[1263] "Means for continuously analyzing user responses" refers to technology that collects user search activity and responses to the results, and optimizes the display of search results from the next time onwards.

[1264] MODE FOR CARRYING OUT THE INVENTION

[1265] This invention is a system that combines a system that collects data from multiple communication tools, stores the data in a database, and analyzes and classifies the data using a natural language processing algorithm with an emotion engine that recognizes the user's emotions. Specific embodiments for implementing this system will be described below.

[1266] 1. Data collection

[1267] The server obtains access rights from multiple communication tools, such as Gmail and chat tools, using API keys and OAuth tokens. It periodically checks for new messages and files using the Gmail API and chat API. For example, when collecting data from Gmail, the server retrieves unread emails and extracts the email content, attachments, sent date and time, and sender information. This extracted data is stored in a database in a data format such as JSON.

[1268] 2. Data Analysis

[1269] The server then analyzes the stored data using natural language processing algorithms, particularly tools like SpaCy and BERT. These tools extract key keywords and context from the content of messages and attachments. This analysis determines the criteria and categories for categorizing the data. The server automatically classifies each message or file based on the extracted keywords and context. It also adds metadata such as timestamps and sender information to complement the data.

[1270] 3. User Emotion Recognition

[1271] When a user accesses the search screen from their device and enters a search term, the server uses an emotion engine to recognize the user's emotion. The emotion engine evaluates the user's emotional state through facial recognition, voice analysis, and text analysis. For example, when a user searches for "Project X Report," the server analyzes the user's voice tone, facial expression, and the tone of the text entered to understand the user's current emotional state.

[1272] 4. Tailor search results based on sentiment

[1273] The server adjusts the display order of search results based on the emotional information recognized by the emotion engine. For example, if the user is feeling stressed, the server will prioritize relevant results and reduce unnecessary information. If the user is feeling relaxed, the server will also display detailed explanations and related materials.

[1274] 5. Search Results Display and Optimization

[1275] The server displays the adjusted search results on the user's device. The user can then review the displayed search results. Furthermore, the server continuously analyzes the user's reactions using an emotion engine to optimize the display of search results from the next time onwards. For example, if a user has a positive reaction to a particular search result, the server will display results based on that reaction when a similar search is performed.

[1276] Examples of concrete examples and prompts

[1277] For example, consider the case where a user searches for "Project X meeting minutes." The server searches the database for messages of meeting minutes related to Project X and sorts them in order of relevance along with metadata such as timestamps and sender information. The emotion engine then adjusts the display order of the search results as needed based on the user's emotional state. The emotion engine also analyzes the user's reaction to the search results and optimizes the display of search results from the next time onwards.

[1278] An example of a prompt sentence might be "Please search for Project X meeting minutes."

[1279] This system allows users to centrally manage what information has been exchanged using which tools, enabling them to efficiently obtain the information they need. In addition, the introduction of an emotion engine makes it possible to provide optimal information according to the user's emotional state, significantly improving work efficiency.

[1280] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1281] Step 1: Collect data

[1282] 1.1 The server obtains API keys and OAuth tokens from multiple communication tools such as Gmail and chat tools. The input is the authentication information of each tool, and the output is the obtained access rights information.

[1283] 1.2 The server uses the obtained access right information to access each tool and periodically checks whether new messages or files exist. The input is the access right information, and the output is the new message or file information.

[1284] 1.3 The server retrieves unread emails using the Gmail API and extracts the email content, attachments, sent date and time, and sender information. The input is access rights information and new message information, and the output is the extracted email data.

[1285] 1.4 The server saves the extracted data in a database in JSON format. The input is the extracted email data, and the output is the data saved in the database.

[1286] Step 2: Analyze the data

[1287] 2.1 The server analyzes the stored data based on a natural language processing algorithm, where the input is the stored data and the output is the analysis result.

[1288] 2.2 The server uses natural language processing tools such as SpaCy or BERT to extract key keywords and context from the content of the message body and attachments. The input is the stored data, and the output is the extracted keywords and context.

[1289] 2.3 The server automatically classifies each message or file based on the extracted keywords and context. The input is the extracted keywords and context, and the output is the classified data.

[1290] 2.4 The server complements the data by adding metadata such as timestamps and sender information. The input is the classified data, and the output is the data with the metadata added.

[1291] Step 3: Recognizing user emotions

[1292] 3.1 The user accesses the search screen from their device and enters a search term. The input is the search term, and the output is the transmission of the search term.

[1293] 3.2 The server passes the user's input to the emotion engine. The input is the search word, and the output is the input to the emotion engine.

[1294] 3.3 The emotion engine evaluates the user's emotional state through facial recognition, voice analysis, and text analysis. The input is the search term and the user's biometric information, and the output is the user's emotional state.

[1295] 3.4 For example, when a user searches for "Project X report," the emotion engine analyzes the user's tone of voice, facial expression, and the tone of the text entered to understand the user's current emotional state. The input is the search word and the user's biometric information, and the output is the analyzed emotional state.

[1296] Step 4: Tailor search results based on sentiment

[1297] 4.1 The server adjusts the display order of search results based on the user's emotional information recognized by the emotion engine. The input is the user's emotional information, and the output is the adjusted search results.

[1298] 4.2 For example, if a user is feeling stressed, the server prioritizes relevant results and reduces unnecessary information. The input is the user's emotional information, and the output is the prioritized search results.

[1299] 4.3 If the user is relaxed, detailed explanations and related materials are also displayed. The input is the user's emotional information, and the output is search results with added detailed information.

[1300] Step 5: Displaying and optimizing search results

[1301] 5.1 The server displays the tailored search results on the user's terminal. The input is the tailored search results and the output is the results displayed on the terminal.

[1302] 5.2 The user can confirm the search results displayed on the terminal. The input is the displayed search results, and the output is the user's confirmation.

[1303] 5.3 The server analyzes user responses to search results (clicks, time spent, etc.) using an emotion engine. The input is the user response, and the output is the analysis result.

[1304] 5.4 The server will optimize the display of search results from the next time onwards based on the analysis results. The input is the analysis results, and the output is the optimized search result display format.

[1305] This series of processing steps allows the system to provide optimal information according to the user's emotional state, allowing the user to efficiently obtain the information they need and improve their work efficiency.

[1306] (Application example 2)

[1307] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1308] In recent years, brick-and-mortar stores have been required to provide services that respond promptly to customer needs, but it is difficult to grasp the emotional state of customers in real time, making it difficult to provide optimal customer service and product recommendations. Furthermore, there is a demand for methods to improve customer satisfaction by utilizing past communication data, but there is a lack of efficient ways to do this. There is an urgent need to provide a system that improves customer experience by resolving these issues.

[1309] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1310] In this invention, the server includes means for collecting data from multiple communication tools, means for saving the collected data in a database, means for analyzing the saved data using a natural language processing algorithm, means for automatically classifying the analyzed data, means for adding metadata to the classified data, means for searching the database based on search words entered by a user on a search screen, means for displaying search results in order of relevance, means for recognizing the user's emotions in real time, and means for adjusting the display order of information based on the recognized emotions. This makes it possible to grasp the emotional state of customers in real time and provide optimal customer service and product suggestions based on past communication data.

[1311] 1. "Communication tools" is a general term for software and hardware used to send and receive information via digital media, such as email and chat applications.

[1312] 2. "Database" means a data management system that efficiently manages collected data and allows for searching and analysis as needed.

[1313] 3. "Natural language processing algorithm" is a general term for algorithms that analyze text data and understand and process its meaning.

[1314] 4. "Classification" is the process of grouping analyzed data based on specific criteria.

[1315] 5. "Metadata" is a general term for attribute information and additional information related to data, which is added to facilitate data search and management.

[1316] 6. "Search screen" is an interface that allows users to enter keywords and search for information in the database.

[1317] 7. "Relevance" is a criterion for sorting search results based on the degree of match or importance with the keywords entered by the user.

[1318] 8. "Emotion recognition means" is a general term for technologies and devices that analyze a user's facial expressions and vocal tone to understand their emotional state.

[1319] 9. "Adjusting the information display order" is the process of changing the order of displayed information based on perceived user sentiment.

[1320] This invention is a system for grasping the emotional state of customers in real time in a physical store and providing optimal customer service and product recommendations. This system collects data from multiple communication tools and analyzes it with a natural language processing algorithm to grasp the user's emotional state in real time and adjust the order in which information is displayed based on this data.

[1321] First, the server collects data from communication tools such as Gmail and chat apps. The server uses the API keys or OAuth tokens of these tools to gain access and periodically checks for new messages and files. The collected data is stored in a database in a data format such as JSON, allowing salespeople to quickly reference past interactions with customers.

[1322] The server then analyzes the stored data using natural language processing algorithms. Generative AI models like SpaCy and BERT are used to extract key keywords and context from the content of messages and attachments. This analysis identifies criteria or categories for classifying the data. Classification tags and metadata are then added to the analyzed data.

[1323] Additionally, the system employs emotion recognition to recognize users' emotions in real time. This is achieved by using the camera and microphone built into the smart glasses to analyze the customer's facial expressions and tone of voice. For emotion recognition, the system uses the Hugging Face Transformers library to evaluate their emotional state. This emotion data is stored in the cloud and can be used to trigger specific actions.

[1324] The system also provides a means to adjust the order in which information is displayed based on emotions. For example, if a customer is feeling stressed, relevant search results will be displayed first, reducing unnecessary information. On the other hand, if the customer is relaxed, detailed explanations and related materials will also be displayed. This adjustment makes it possible to provide the optimal service according to the customer's state.

[1325] The following specific examples are given as examples.

[1326] Example 1: When a customer searches for "Project X meeting minutes"

[1327] The server searches for meeting minutes related to Project X and displays them in order of relevance along with timestamps and sender information. The emotion engine adjusts the display order according to the user's emotional state and provides detailed information as needed.

[1328] Example 2: When a customer is stressed

[1329] Prompt: "What is the appropriate way to respond when a customer is stressed?"

[1330] Example output: "Respond simply and quickly, and refrain from suggesting new products."

[1331] Example 3: When the customer is relaxed

[1332] Prompt: "What is the appropriate way to respond when a customer is relaxed?"

[1333] Sample output: "Provides detailed product descriptions and offers additional options."

[1334] This will enable customer service in physical stores to respond more quickly to the emotional state of customers and provide greater satisfaction.

[1335] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1336] Step 1:

[1337] The server obtains access rights using API keys or OAuth tokens from communication tools such as Gmail and chat apps. The input is the authentication information for each communication tool, and the output is the access rights to these tools. Specifically, the server periodically checks for and collects unread emails using the Gmail API.

[1338] Step 2:

[1339] The server collects new messages and files from the communication tools it has gained access to and stores them in a database in an appropriate format, such as JSON. The input is the data of the new messages and files, and the output is the data stored in the database. Specifically, timestamps and sender information are also stored in the database at the same time.

[1340] Step 3:

[1341] The server analyzes the stored data using natural language processing algorithms (e.g., SpaCy or BERT). The input is the text data stored in the database, and the output is the analysis results, including key keywords and contextual information. Specifically, meaning is extracted from the contents of the message body and attachments and automatically classified.

[1342] Step 4:

[1343] The server adds classification tags and metadata to the analyzed data. The input is the main keywords and context information from the analysis results, and the output is data with the classification tags and metadata added. Specifically, category information, timestamp, and sender information are added to each data.

[1344] Step 5:

[1345] A user enters keywords into a search screen, and the server searches the database based on those keywords. The input is the search keyword entered by the user, and the output is a list of related data. Specifically, the server extracts highly relevant data from the searched data.

[1346] Step 6:

[1347] The server sorts the search results in order of relevance and displays them to the user. The input is a list of searched data, and the output is the search results sorted in order of relevance. Specifically, the list is displayed taking into account related metadata.

[1348] Step 7:

[1349] The system grasps the user's emotional state in real time through a camera and microphone connected to a device (such as smart glasses). The input is camera video and microphone audio, and the output is the analysis result of the user's emotional state. Specifically, it uses the Hugging Face Transformers library to recognize emotions from facial expressions and voice tone.

[1350] Step 8:

[1351] The server adjusts the display order of information based on the recognized emotional state. The input is the analysis result of the user's emotional state and the search results, and the output is the adjusted display order of information. Specifically, if the user is feeling stressed, highly relevant information is displayed first, and if the user is relaxed, additional detailed information is displayed.

[1352] Step 9:

[1353] The server continuously analyzes customer responses using an emotion engine to optimize the display of search results from the next time onwards. The input is customer response data, and the output is search results in an optimized display order. Specifically, the next display is improved based on search results that have generated positive responses.

[1354] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1355] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1356] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1357] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1358] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1359] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1360] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1361] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1362] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1363] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1364] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1365] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1366] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1367] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1368] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1369] The hardware resource for executing a specific process can be any of the following types of processors: A processor, for example, is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1370] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1371] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1372] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1373] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1374] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1375] The following is further disclosed regarding the above embodiment.

[1376] (Claim 1)

[1377] A means of collecting data from multiple communication tools;

[1378] a means for storing the collected data in a database;

[1379] A means for analyzing the stored data using natural language processing algorithms;

[1380] means for automatically classifying the analyzed data;

[1381] means for attaching metadata to the classified data;

[1382] A means for searching the database based on search words entered by the user on the search screen;

[1383] a means of displaying search results in order of relevance;

[1384] A system including:

[1385] (Claim 2)

[1386] 10. The system of claim 1, further comprising: means for obtaining access to the database.

[1387] (Claim 3)

[1388] 10. The system of claim 1, further comprising means for extracting file, timestamp, and originator information from the collected data.

[1389] "Example 1"

[1390] (Claim 1)

[1391] a means for collecting data from multiple communication tools;

[1392] means for storing the collected data in a data management device;

[1393] A means for analyzing the stored data using natural language processing algorithms;

[1394] means for automatically classifying the analyzed data;

[1395] means for attaching metadata to the classified data;

[1396] A means for searching the data management device based on a search word input by an information user on a search screen;

[1397] a means for displaying search results in order of relevance;

[1398] Using natural language processing algorithms such as SpaCy and BERT,

[1399] A means of storing the collected data in JSON format;

[1400] A means of regularly checking for new messages and files from your communication tools;

[1401] A system including:

[1402] (Claim 2)

[1403] 10. The system of claim 1, further comprising: means for obtaining access rights to the data management device.

[1404] (Claim 3)

[1405] 10. The system of claim 1, further comprising means for extracting data objects, time information, and originator information from the collected data.

[1406] "Application Example 1"

[1407] (Claim 1)

[1408] a means for collecting data from multiple communication means;

[1409] means for storing the collected data in an information storage device;

[1410] A means for analyzing the stored data using natural language processing algorithms;

[1411] means for automatically classifying the analyzed data;

[1412] a means for adding attribute information to the classified data;

[1413] means for searching the information storage device based on a search instruction input by a user on a search screen;

[1414] a means of displaying search results in order of relevance;

[1415] a means of providing information regarding the maintenance of factory equipment;

[1416] A system including:

[1417] (Claim 2)

[1418] 10. The system of claim 1, further comprising means for obtaining access to the information storage device.

[1419] (Claim 3)

[1420] 10. The system of claim 1, further comprising means for extracting document, time, and sender information from the collected data.

[1421] "Example 2: Combining Emotion Engines"

[1422] (Claim 1)

[1423] A means of collecting data from multiple communication tools;

[1424] a means for storing the collected data in a database;

[1425] A means for analyzing the stored data using natural language processing algorithms;

[1426] means for automatically classifying the analyzed data;

[1427] means for attaching metadata to the classified data;

[1428] A means for searching the database based on search words entered by the user on the search screen;

[1429] a means of displaying search results in order of relevance;

[1430] a means for using an emotion engine to recognize a user's emotion when accessing a search screen;

[1431] a means for adjusting the display order of search results based on the emotional information recognized by the emotional engine;

[1432] A method to continuously analyze user reactions using an emotion engine and optimize the display of search results from the next time onwards.

[1433] A system including:

[1434] (Claim 2)

[1435] 10. The system of claim 1, further comprising: means for obtaining access to the database.

[1436] (Claim 3)

[1437] 10. The system of claim 1, further comprising means for extracting date, time information, and caller information from the collected data.

[1438] "Application example 2 when combining emotion engines"

[1439] (Claim 1)

[1440] A means of collecting data from multiple communication tools;

[1441] a means for storing the collected data in a database;

[1442] A means for analyzing the stored data using natural language processing algorithms;

[1443] means for automatically classifying the analyzed data;

[1444] means for attaching metadata to the classified data;

[1445] A means for searching the database based on search words entered by the user on the search screen;

[1446] a means of displaying search results in order of relevance;

[1447] a means for recognizing user emotions in real time;

[1448] means for adjusting the display order of information based on the recognized emotion;

[1449] A system including:

[1450] (Claim 2)

[1451] 10. The system of claim 1, further comprising: means for obtaining access to the database.

[1452] (Claim 3)

[1453] 10. The system of claim 1, further comprising means for extracting file, timestamp, and originator information from the collected data. [Explanation of symbols]

[1454] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of collecting data from multiple communication tools; a means for storing the collected data in a database; A means for analyzing the stored data using natural language processing algorithms; means for automatically classifying the analyzed data; means for attaching metadata to the classified data; A means for searching the database based on search words entered by the user on the search screen; a means of displaying search results in order of relevance; A system including:

2. 10. The system of claim 1, further comprising means for obtaining access to the database.

3. 10. The system of claim 1, further comprising means for extracting file, timestamp, and originator information from the collected data.

Images